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FOREWORD 



This report is a product of the National Postsecondary Education Cooperative (NPEC). It was 
undertaken as part of NPEC’s ongoing efforts to explore various dimensions of postsecondary 
education outcomes. 

Twelve individuals, each of whom had an extensive background in competency-based 
postsecondary education but in vastly different settings, were brought together. Their charge was 
to develop a project in the area of competency-based education that would be of value to a wide 
spectrum of the postsecondary education community. At its first meeting, this Working Group 
concluded that since competency-based initiatives were relatively new in postsecondary 
education, the most useful task they could undertake would be to develop a basic guide to 
postsecondary competencies. This guide would be intended for educators who were unfamiliar 
with the field but who might have an interest in establishing a competency-based initiative at their 
postsecondary institution. This report is the result of that undertaking. 

It contains an annotated bibliography that introduces a novice in the field to basic materials about 
the construction and use of competencies. It includes the results of eight case studies of 
competency-based postsecondary programs that were identified by members of the Working 
Group as exemplifying diverse but interesting approaches. It also contains a set of operating 
principles that were gleaned by the Working Group and its consultants from commonalities 
among the eight case studies. 

This report was reviewed and accepted by the NPEC Steering Committee as meeting NPEC’s 
standards and policies. We hope users of this document will find it helpful as they begin or 
continue their explorations into competency-based postsecondary education. 
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EXECUTIVE SUMMARY 



This document examines the use of competency-based initiatives across postsecondary education 
in the United States and presents principles that underlie successful implementation drawn from 
selected case studies. Conducted under the auspices of the National Postsecondary Education 
Cooperative, this project was informed by a Working Group of individuals selected for their 
expertise in utilizing competencies in a variety of settings. This project began in September 1998 
and concluded in October 2000. 

The world is changing and so is postsecondary education. Access to learning opportunities is 
greater now than at any previous time (Rosenberg, 2000; Voorhees, 2001). The learning paths 
created by advances in information technology no longer lead solely to postsecondary 
institutions. Organizations outside of postsecondary education have made significant inroads by 
providing performance-based learning opportunities built on competencies. It is now possible for 
more sophisticated education consumers, (i.e., students and potential students) to acquire skills 
and competencies through various means and at times they prefer. The Wingspread Group on 
Higher Education (1993, p. 14) noted that, “putting learning at the heart of the academic 
enterprise will mean overhauling the conceptual, procedural, curricular, and other architecture of 
postsecondary education on most campuses.” Some colleges and universities have undergone 
dramatic transformations in response to this movement toward performance-based learning, while 
others have maintained a traditional curriculum packaged in standard delivery formats and 
provided in regular academic terms. 

Competencies are the bridge between traditional credit hour measures of student achievement and 
the learning revolution. For the purposes of this project, a competency is defined as “a 
combination of skills, abilities, and knowledge needed to perform a specific task.” “Competency- 
based initiatives,” then, are those purposeful actions undertaken by postsecondary institutions 
directed at defining, teaching, and assessing competencies across their system. 

Why is it important to implement competency-based initiatives in colleges and universities? One 
main reason is that specific articulations of competencies inform and guide the basis of 
subsequent assessments at the course, program, and institutional levels. Secondly, specific 
competencies help faculty and students across campus, as well as other stakeholders such as 
employers and policymakers, to have a common understanding about the specific skills and 
knowledge that undergraduates should master as a result of their learning experiences. Assuming 
that faculty use a formal process to get feedback about what the competencies should be, then 
stakeholders are more likely to accept and value them. Third, specific competencies provide 
directions for designing learning experiences and assignments that will help students gain practice 
in using and applying these competencies in different contexts. 

A major challenge for faculty, staff, and administrators is to ascertain the “data ramifications” 
that ensure that competencies are both valid and reliable with the ultimate aim, in practice, being 
that they are fully transportable between and outside of postsecondary entities. Data 
ramifications also refer to efforts to describe competencies in a uniform manner so that they can 
have the same meaning in a variety of contexts and for a variety of audiences. If competencies 
are to be utilitarian, strategies must be built to ensure that they are uniform. It is this 
standardization of terminology and semantics that can lead to the transportability of competencies 
among sectors of postsecondary education and across organizations that are not immediately 
aligned with colleges and universities. 
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Disappointingly, few sites in this study were actively dealing with issues of reliability and 
validity. Instead, as might be expected of relatively new endeavors, issues of process and internal 
acceptance for competencies were the initial focus taking time, energy, and resources that might 
otherwise (with greater experience) be used to tackle transportability. In the future, as 
organizations and postsecondary institutions gain more experience with competency-based 
initiatives, a focus on data ramifications will be important. By attending to concerns about 
validity and reliability, institutions can glean meaningful information to improve their initiatives 
and to satisfy external demands for accountability. 

Competencies are crucial for students before, during, and after their attendance at postsecondary 
institutions. Consequently, these points in time were chosen as the organizing framework for this 
work* as shown in exhibit E-l. This project examined how competencies were used in various 
settings (including eight case studies of postsecondary institutions and a corporation) at key 
points in a student’s education. 

The ultimate goal of this project was to produce a final report that could serve as a hands-on 
resource for practitioners who seek to develop, implement, or refine their competency-based 
initiative. Accordingly, the case studies and the expertise of the Working Group were drawn 
upon to synthesize and recommend the following principles for practice in postsecondary 
education. While each case study yielded fruitful insights, there are common practices gleaned 
from this study that can enhance the likelihood of success for those wishing to embark on 
competency-based initiatives. 

• A senior administrator is the public advocate, leader, and facilitator for creating an 
institutional culture that is open to change, willing to take risks, and fosters 
innovations by providing real incentives for participants. 

• The appropriate stakeholders fully participate in identifying, defining, and reaching a 
consensus about important competencies. 

• Competencies are clearly defined, understood, and accepted by relevant stakeholders. 

• Competencies are defined at a sufficient level of specificity that they can be assessed. 

• Multiple assessments of competencies provide useful and meaningful information 
that is relevant to decision-making or policy-development contexts. 

• Faculty and staff fully participate in making decisions about the strongest assessment 
instruments that will measure their specific competencies. 

• The precision, reliability, validity, credibility, and costs are all considered and 
examined in making selections about the best commercially developed assessments 
and/or locally developed approaches. 



This temporal sequence complements the “across provider” framework offered by the National Postsecondary 
Education Cooperative Working Group on Student Transitions. That framework examines student movement across 
multiple providers and assumes that competencies are adding value as a transportable unit. 



viii 



1 3 BEST copy available 



Exhibit E-l. — Framework for organizing competency-based initiatives 



Institutions and organizations | 


Ford Motor 
Company 




















Proficiency- 
based Admission 
Standards System 
(PASS) in 
Oregon 




X 


X 














Western 

Governors 

University 










X 










Colorado 
Community 
Colleges 
Incumbent 
Worker Project 










X 










Hagerstown 

Community 

College 

(National Science 
Foundation 
Consortium) 










X 










Sinclair 

Community 

College 










X 










Northwest 
. Missouri State 
University 










X 


X 


X 


X 




King’s College 










X 


X 








Key transition 


Entry into postsecondary 
education (PSE) 


■ Competency-based 
secondary school 
graduation 


■ Competency-based 
admissions 


| Within PSE I 


■ Competency-based 
curricula in specific 
disciplines 


■ General education 

competencies 


■ Competencies at transfer 

(within providers) 


■ Competencies at transfer 

(across providers) 


■ Competencies certified 

through standardized 
tests 




ix 14 



BEST COPY AVAILABLE 



T3 

OJ 

3 



o 

u 

C/3 

O 

> 



-a 

a> 

C/3 

03 

XI 

i 

>> 

o 

C 

a> 

a> 

a 

E 

o 

u 

W) 



03 

OD 

u 

o 

u 

£ 

u 

© 

£ 

CJ 

E 

03 

u 

fa 



fa 

"M 

3 



X 

fa 



Institutions and organizations | 


Ford Motor 
Company 














Proficiency- 
based Admission 
Standards System 
(PASS) in 
Oregon 














Western 

Governors 

University 




X 










Colorado 
Community 
Colleges 
Incumbent 
Worker Project 














Hagerstown 

Community 

College 

(National Science 
Foundation 
Consortium) 














Sinclair 

Community 

College 














Northwest 
Missouri State 
University 




X 






X 


X 


King’s College 




X 






X 




Key transition 


| Exit from PSE 


■ End of program 

competencies 


■ PSE to employment 

competencies 


Overall institutional 
effectiveness 


■ Internal (improvements 

in student learning and 
academic programs) 


■ External (accountability, 

performance budgeting, 
accreditation) 




x 



15 



• The competency-based educational initiative is embedded within a larger institutional 
planning process. 

• The assessments of competencies are directly linked with the goals of the learning 
experience. 

• The assessment results are used in making critical decisions about strategies to 
improve student learning. 

• The assessment results are clear and reported in a meaningful way so that all relevant 
stakeholders fully understand the findings. 

• The institution experiments with new ways to document students’ mastery of 
competencies that supplement the traditional transcript. 

Each principle is more fully delineated in section 4 and accompanied by illustrations from 
specific case studies. Research into competencies has not been reported extensively in the higher 
education literature. Nevertheless, there are useful and thoughtful resources about how to 
implement competencies. Examples of recent literature are annotated and presented in the 
bibliography in section 6. 
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1. INTRODUCTION 



A competency is a combination of skills, abilities, and 
knowledge needed to perform a specific task in a 
given context. 

In 1997, a National Postsecondary Education Cooperative (NPEC) Working Group examining 
data needs related to workforce development policies noted an increased interest nationally in 
skill standards, competency-based credentials, and assessment of work-based learning 
experiences. NPEC’s mission is “to promote quality, comparability and utility of postsecondary 
data and information that support policy development” (Public Law 103-382, Section 410, 
Improving America’s Schools Act of 1994). This Working Group recommended the 
establishment of a NPEC project that would look at these emerging trends from the context of 
data and data ramifications. Advances in educational technology making access to competency- 
based learning more prevalent combined with competition from groups other than traditional 
postsecondary providers accelerated interest in this topic. In this context, performance-based 
learning has gained increasing attention. 

There are numerous challenges associated with developing and assessing competency-based 
initiatives based upon performance standards. Faculty and staff must decide upon the 
competencies to examine and levels of performance. They deliberate about the best 
methodologies to assess student performance, including whether to use locally developed 
instruments or commercially available standardized tests. They decide who will be responsible 
for assessments and how the results will be used. It is not clear how problems related to 
portability of competency assessment and credentials across states and institutions will be 
resolved. These issues have ramifications for new data priorities as states, educational 
institutions, and training providers encounter uncharted territory in developing performance 
standards and assessing competencies. 

A number of states and institutions are currently supporting competency-based initiatives, and 
efforts to define critical skills have been underway for some time at the federal level, as well as 
under the auspices of the national testing services. Even so, there is no comprehensive source for 
determining the extent to which postsecondary entities are engaged in competency-based 
initiatives and the extent that issues of transportability across a wide spread of providers are 
playing into that work. At its meeting in January 1998, the NPEC Steering Committee approved 
a new Working Group and asked it “to analyze current practices in defining and assessing learner 
competencies and to determine their utility across a range of contexts.” The key tasks assigned to 
the Working Group were to: 

• Address the current state of the use of competencies across postsecondary education; 

• Investigate and report the data ramifications of competency-based initiatives in 
postsecondary institutions and their connections to student learning; 

• Demonstrate the importance of competency-based initiatives in broader contexts such 
as economic development, technology-based instruction, and industry interests in 
competency certifications; 

• Address terminology and definitional issues: What is a competency? How does it 
differ from a student outcome, skill, or ability? What is the relationship between 
performance, assessment, and standards? 
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• Reach out to important target audiences (e.g., presidents, board members, legislators) 
that are not likely to avail themselves of other work in the area of competency 
definition and assessment; and 

• Establish a set of “strong principles” or key considerations in the decision-making 
process for defining and measuring competencies. 



Members of the Working Group brought wide and focused expertise to bear on these tasks, 
including: 

• Developing competency-based admissions processes in higher education; 

• Linking skills with job requirements; 

• Developing career transcripts; 

• Assessing critical thinking and writing skills; 

• Defining workforce development performance measures; and 

• Carrying out assessments at different points in the learning process. 



It should come as no surprise that these areas are ubiquitous in the wide arena of competency- 
based initiatives. It is very challenging to create, plan, and implement competency-based 
systems. In many instances, major political processes both within postsecondary education and in 
the external environment dictate the will to implement them and the success that they might 
achieve. The Working Group discussed purposes, principles, pitfalls, and processes for defining 
and assessing competencies within these areas, and concluded that future directions should 
involve an examination of the state of current practices through a case study approach. The 
resulting case studies form the foundation for gleaning the principles of effective practice in 
competency-based initiatives presented in section 8 of this document. 

The Working Group also struggled with the semantically sensitive issues that are attendant to 
efforts in this area. For example, what is considered a “competency” in one setting may be 
regarded as an “objective” or “skill” in another setting. There is no single definition of a 
competency that cuts across all contexts, necessitating development of an operational definition 
to guide this project. 

The Working Group also was cognizant of its relationship to other Working Groups 
commissioned by NPEC. By casting its definition of competency, in examining state-of-the-art 
practice, and in making recommendations for future practice, it is hoped that this work can inform 
not only NPEC but the wider postsecondary universe as well. For example, the conceptual 
framework, with its emphasis on times in a postsecondary education career that competencies are 
important, parallels the efforts of another NPEC Working Group (Student Transitions Working 
Group) entitled “Examination of Policy Issues and Data Needs at Points of Student Transition” 
(Student Transition Data Systems: Phase I Report of the NPEC Working Group, November 
2000 ). 
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A Quick Roadmap to This Report 

Following this introduction (section 1), the report is divided into seven sections. The following 
“roadmap” may aid the reader to use this document efficiently. 

• Section 2: Provides a conceptual overview of competencies as implemented in a 
variety of settings, including key concepts and definitions, which can be useful as a 
starting point for institutions that previously have not considered competencies and 
their connection to the learning process. 

• Section 3: Establishes the methodology used by the Working Group to select sites for 
the case study investigation. Gives practitioners a quick overview of the criteria used 
to determine “cutting-edge” case study locations, as well as a glimpse of key issues 
probed during interviews. 

• Section 4: Provides a quick read on the overarching principles and strong practices 
uncovered within the case studies. Helps institutions, state agencies, and 
policymakers learn about how to improve or create competency-based initiatives with 
examples from other settings. 

• Section 5: Outlines an agenda for future work and a call for action at the institutional 
level. 

• Section 6: Offers a current, annotated bibliography of resources in the literature 
about competencies. Serves as a guide to the best resources for both practitioners and 
policymakers. 

• Section 7: Contains abstracts of the eight case studies. 

• Section 8: Appendices A-H present the eight case studies for those practitioners who 
want more details and analyses of current practices in assessing and utilizing 
competencies within a specific institutional or corporate context. Appendix I is the 
case study interview protocol. 
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2. DEFINING AND CLASSIFYING COMPETENCY-BASED INITIATIVES: 

THE GUIDING FRAMEWORK 



The challenge is to determine which competencies can be bundled 
together to provide which types of learners the optimal 
combination of skills and knowledge needed to perform a 
specific task. 

What is a competency? Why is it important to assess competencies? How are assessments of 
competencies used in making decisions and developing policy? The Working Group’s 
deliberations on these topics are presented in this section. The Working Group relied upon the 
wide range of experiences reflected in its membership to develop a first-cut definition of the 
scope of competency-based initiatives and used that pragmatically derived definition to guide the 
compilation of an annotated bibliography of competency-based initiatives. 

The world grows more complex on a daily basis as knowledge increases dramatically each year, 
and access to learning opportunities is greater now than at any previous time. Information 
technology has expanded learning paths that do not lead solely to postsecondary institutions. The 
interest in learning “anywhere, anyplace, anytime” means that traditional educational delivery 
systems will be challenged to address the needs of diverse learners. Along with expanded access, 
potential students and undergraduates will likely become more sophisticated consumers and will 
grasp the concept of acquiring skills and competencies through diverse means. The Wingspread 
Group on Higher Education (1993, p. 14) noted that, “putting learning at the heart of the 
academic enterprise will mean overhauling the conceptual, procedural, curricular, and other 
architecture of postsecondary education on most campuses.” This report is not intended, by 
itself, to change the culture of most postsecondary institutions. However, it can offer guideposts 
and examples of good practice for those institutions and entities that want to become more fully 
engaged in performance-based learning. Ultimately, the process of implementing competencies 
can be a valuable lever for creating new opportunities and collaborations among postsecondary 
institutions and other organizations. 

The case study findings reported here explore the beginning work of several initiatives that are in 
the early stages of development. A couple of sites have more extensive experience. Collectively, 
the studies provide strong evidence that most competency-based initiatives are at the embryonic 
stage of development across postsecondary education in the United States. In contrast, Great 
Britain has created a skills system under the auspices of the National Learning and Skills Council 
(Betts and Smith, 1998). The Council operates through 47 local Learning and Skills Councils 
and is supported by over 100 local Learning Partnerships. Whereas the early returns from this 
experience indicate that this system is somewhat inflexible and perhaps not as immediately 
responsive to marketplace needs as it might be, its establishment at least provides the foundation 
for the transportability of skills across sectors. It is not clear whether the recent push toward 
competency-based initiatives on behalf of a handful of institutions in the United States will ever 
approach a national system similar to Great Britain’s. The current architecture of higher 
education in the United States does not easily promote the open exchange of learner 
competencies across sectors (e.g., from community colleges to 4-year institutions across states). 
In the meanwhile, institutions and students are often left to navigate issues of transportability of 
learning experiences in uncharted waters. 

During initial meetings, the Working Group utilized its collective experiences and reviewed an 
initial draft of a bibliography prepared by Karen Paulson, consultant to the Working Group. The 
Working Group then developed a framework for classifying competency initiatives. This 
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framework consists of four major stages where competencies are important: (1) entry into 
postsecondary education, (2) points of transition within postsecondary education, (3) exit from 
postsecondary education, and (4) the overall effectiveness of postsecondary education. These 
elements provide a broad framework (as outlined below) for conceptualizing competency-based 
initiatives both within and outside of postsecondary institutions. The Working Group identified a 
subset of these elements for further examination through case studies. 



Entry Into Postsecondary Education 

• Competency-Based Secondary School Graduation 

• Competency-Based Certification of Skill Attainment 

• Competency-Based Admissions to Postsecondary Institutions 

• Competency-Based Placement in Classes 

• Competency-Based Military Experiences 

Within Postsecondary Education 

• Competency-Based Curricula in Specific Disciplines 

• General Education Competencies 

• Competencies at Transfer Within a Single Postsecondary Education Provider (e.g., 
change of majors or program of study) 

• Competencies at Transfer Across Multiple Postsecondary Education Providers (e.g., from 
one institution to another) 

• Competencies Certified Through Standardized Testing 



Exit From Postsecondary Education 

• End of Program Competencies 

• Postsecondary to Employment (Workplace Competencies: Demonstrations on the Job) 

• Postsecondary to Employment Eligibility (Licensures, Certifications, or Board 
Examinations) 

• Postsecondary to Graduate/Professional School 

• Lifelong Learning 

Overall Institutional Effectiveness 

• Internal (Improvements in Student Learning and Academic Programs) 

• External (Accountability, Performance Budgeting, Accreditation) 



O 

ERIC 



6 



r 2S 



This framework was used as the basic classification structure for the annotated bibliography and 
for selecting the limited number of competency-based initiatives to study. This structure also is 
useful for defining links across the activities carried out within the project and ultimately among 
the products resulting from this work. This structure also parallels the work of the NPEC 
Working Group on Student Transitions. 1 This particular NPEC Working Group represents a 
similar effort to understand learner mobility. Both groups endorse the premise that learning 
pathways from school to college and to work can no longer be defined in terms of highly 
structured, linear patterns and timeframes. Rather, learners (during the course of a lifetime) are 
likely to pursue many different transitions between learning experiences and work, and between 
further training and additional education. Gaining insights into these transitions is important so 
that the documentation of learning (via competencies) can lead to smoother student transitions 
within and outside of postsecondary education. 



Key Concepts and Definitions 

Many terms are used interchangeably to describe learners and the results of the learning process, 
including outcomes, skills, traits, characteristics, competencies, and domains. Although the 
boundaries and meanings among these words are frequently blurred, a clear definition of what is 
meant by competency is needed to guide this work. The ultimate definition of competency as 
used by the Working Group and throughout this document is the combination of skills, abilities, 
and knowledge needed to perform a specific task. Figure 1 was adopted to provide demarcation 
among terms and to assist others to visually differentiate the hierarchy chosen by the Working 
Group. 



• Traits and Characteristics are the foundation for learning, the innate make-up of 
individuals on which further experiences can be built. Differences in traits and 
characteristics help explain why people pursue different learning experiences and 
acquire different levels and kinds of knowledge and skills. 

• Skills, Abilities, and Knowledge are developed through learning experiences, 
broadly defined to include school, work, participation in community affairs, etc. 
(although the Competency-Based Initiatives project is focusing on formally 
organized postsecondary education learning processes). 

• Competencies are the result of integrative learning experiences in which skills, 
abilities, and knowledge interact to form bundles that have currency in relation to the 
task for which they are assembled. 

• Demonstrations are the results of applying competencies. It is at this level that 
performance can be assessed. 



1 For more information on the Student Transitions Working Group, as well as the other NPEC initiatives, visit the 
NPEC Web Site at http://nces.ed.gov/npec/projects.asp 
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Figure 1. — A hierarchy of postsecondary outcomes 




Characteristics of Competencies 

Skills, abilities, knowledge, and competencies are often used interchangeably by those within and 
outside of postsecondary education. For example, we speak of competent machinists and highly 
skilled machinists; competent mathematicians and knowledgeable mathematicians. The hierarchy 
in figure 1 seeks to differentiate these terms: skills and knowledge are acquired through learning 
experiences; different combinations of skills and knowledge that one has acquired define the 
competencies that an individual possesses. Finally, different combinations of competencies 
possessed by an individual are combined in carrying out different demonstrations or tasks. 

Administrators, faculty, and policymakers frequently use the term “outcomes of postsecondary 
education.” At a basic level, such language would seem to encompass the skills and knowledge 
acquired in the learning process. However, and perhaps because of its wide usage in a variety of 
contexts, the term also includes, for many, psychosocial development; attitudes, values and 
beliefs; and civic development (see, for example, table 2: A Taxonomy of Student Outcomes, 
U.S. Department of Education, 1997.) Widely used, the term “outcomes” also encompasses 
proxies for skills, knowledge, and abilities, including the completion of a degree or course, 
grades, certifications, employment, and salary. Each of these signposts is measurable, 
attributable in varying degrees to postsecondary experiences, and time-linked. A fuller scope of 
postsecondary outcomes is directly related across an individual’s lifespan, including better health 
and the economic benefits passed on to his/her children. 

In this view, then, competencies are complementary phenomena. They are a combination of 
skills, abilities, and knowledge. Knowledge of the relationship between voltage, resistance, and 
amperage (i.e., knowledge of Ohm’s Law, whether put in those terms or not) and skills in 
soldering and splicing wires are among those needed to be a competent electrician. These 
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competencies can lead to certification and eventual employment. They also can contribute to a 
broader set of postsecondary outcomes, including economic return to self and society. 

A single competency can be used in many different ways. For example, measuring distances is 
important to both professional golfers and surveyors. Of course, different measuring skills may 
be involved in carrying out these two tasks, but the skill involved in performing measurement, 
irrespective of technique or method, should produce the same result. It is in their context, 
however, that competencies have their greatest utility. 

Competencies within different contexts require different bundles of skills and knowledge. It is 
precisely this “bundling” and “unbundling” that drives competency-based initiatives among 
postsecondary entities. The challenge is to determine which competencies can be bundled 
together to provide which types of learners the optimal combination of skills and knowledge 
needed to perform a specific task. Leadership in a surgery suite is different from leadership on 
the basketball court. For example, motivating teammates is more important to leadership in 
basketball, while superior knowledge of the procedure is more important to leadership in surgery. 
In both contexts, however, an ability to effectively coordinate the roles, timing, and contributions 
of coworkers is critical. The bundles of different skills and knowledge that are given the same 
label in different contexts is one reason there is often difficulty in achieving a common 
understanding of what a given competency (like leadership) is, and then what it means to assess 
it. 

Knowing how to package the right set of competencies to effectively carry out a given task is in 
itself a competency. We sometimes refer to individuals as having great skills, but these same 
people appear unable to apply them. With experience and experimentation, people combine 
gestures, phrases, eye contact, pace of speech, etc., in ways that allow them to give better 
speeches. It is easy to see that maturation, motivation, and opportunities to practice are keys to 
understanding the bundling and unbundling processes. 



Using Competencies 2 

Competency-based initiatives seek to insure that students attain specific skills, knowledge, and 
abilities considered important with respect to whatever they are studying or the transitions for 
which they are preparing. Utilizing competencies requires the development of three distinct, but 
interactive components: 

• A description of the competency; 

• A means of measuring or assessing the competency; and 

• A standard by which someone is judged to be competent. 

The following examples illustrate these components in selected areas within the study’s 
classification structure: 



2 This section draws heavily upon a communication to the Working Group from Joy McClarty, former member of the 
NPEC Competency-Based Initiatives, in March 1999. 
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Entry into postsecondary education (competency-based admissions) 

• Defining competencies: often done by curriculum panels of faculty and teachers; 

• Assessing competencies: multiple methods are used, including standardized tests, 
teacher ratings using their own scoring guides, and evaluations of student work 
samples or portfolios; and 

• Standards forjudging competence: often set by a master panel of secondary and 
postsecondary faculty. 

Exit from postsecondary education — postsecondary education to employment 

(certification and licensure) 

• Defining competencies: usually done by members of the profession with a “public 
protection” perspective; 

• Assessing competencies: various methods used including licensing exams, 
completion of accredited programs, and successful experience in specified levels of 
practice; and 

• Standards forjudging competence: established in practice, laws, or certifications that 
have credibility with employers. 



Competency-based initiatives can be important in communicating to students which competencies 
are important for them to attain and the extent to which their learning experiences/efforts are 
meeting these expectations (i.e., the student as a consumer of information about competencies). 

Competency-based initiatives also can be important in communicating to employers or the 
general public what people know and are able to do. Thus, a primary advantage of using 
competency definitions, assessments, and standards is the process of identifying which 
competencies are important for students to attain, and the quality assurance that students actually 
attain them. 

When issues of transporting competencies arise, it is even more critical that they be described 
both qualitatively and quantitatively in a common language set. It is in this domain that the 
Working Group devoted considerable energy. Difficulties are created for students and providers 
when they use the same term for different competencies; or conversely, when different standards 
for assessing the same competencies are employed. Both undermine efforts to develop a common 
language and subsequently the utility of competency-based initiatives as a means of 
communicating with students, employers, and the general public. Unfortunately, there is 
currently no common, utilitarian rubric other than the framework offered in this final report that 
can address the tendency by providers and other groups to develop their own definitions and 
subsequent standards for assessing competencies. Lack of national consensus on definitions and 
standards by which competencies can be judged impedes their ultimate transportability across a 
variety of contexts. 
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Data Ramifications of Competency-Based Initiatives 

What separates this work from other efforts to understand competencies and their position in the 
constellation of postsecondary education? The charge to the Working Group was “to analyze 
current practices in defining and assessing learner competencies and to determine their utility 
across a range of contexts.” It is this last word, “utility,” that serves as a demarcation point for 
this project, underscoring the need to understand the data ramifications of competencies for 
administrators, policymakers, and postsecondary institutions. 

What are data ramifications and why should anyone care? This topic invites memories of 
graduate programs in statistics, where data and decisions about data are paramount. How can 
data be tied to competencies, or what do numbers have in common with outcomes, skills, 
attributes, and objectives? Beyond the numerical realm and in a broad sense, the term “data 
ramifications” refers to efforts to describe competencies in a uniform manner so that they have 
the same meaning in a variety of contexts. It is this standardization of terminology and semantics 
that can lead to the transportability of competencies across different levels of education among 
postsecondary entities and external stakeholders such as employers. 

The statistical concepts most aligned with data ramifications in this arena, especially in 
considering the transportability of competencies, are reliability and validity. For reliability, 
faculty and policymakers are interested in whether a measurement technique, applied repeatedly 
and across the same subjects, will yield the same result each time. For institutions, this becomes 
an issue of quality control in choosing whether to bundle or unbundle a competency earned or 
certified by another provider. 

The concept of validity strikes somewhat deeper meaning in an assessment context. That is, the 
main concern is whether the measure of a competency really reflects the true meaning of that 
competency. At a surface level, this is referred to as “face validity.” That is, does the 
measurement of a competency reasonably have anything to do with the understanding of that 
competency. In a more direct way, we also refer to “predictive validity,” or referencing an 
assessment to an external criterion such as admission to college or success as a tool and die 
operator. “Content validity,” in contrast, refers to whether an assessment is sufficiently inclusive 
of the competency at hand. For example, a mathematical computation competency test should 
not stop with just addition, but should also include subtraction, multiplication, and division. 
Finally, “construct validity” refers to the way in which an assessment or measure relates to other 
variables within a system of theoretical relationships. For instance, there might be interest in how 
a measure relates to “overall undergraduate satisfaction.” 

It is important to determine the validity and reliability of both formative and summative 
assessments. Faculty and staff typically conduct formative assessments during the life of an 
academic program. The main purpose is to provide feedback to stakeholders that can be used in 
making key decisions about the best way to modify, shape, and ultimately improve the academic 
program and student learning (Palumbo and Banta, 1999). If competencies are clear, meaningful, 
and fully understood by relevant stakeholders, there is greater likelihood that linkages across 
academic programs and even across institutions within a state or consortium can be created. Such 
linkages create smoother transitions for students who move between institutions and also provide 
greater coherence in the curriculum. If stakeholders are satisfied with the levels of reliability and 
validity of the assessment methods as they review results, then such approaches will have greater 
credibility as students seek to get appropriate courses waived or receive academic credit for 
previous learning experiences. 
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Data ramifications are also directly aligned with external demands for accountability. Faculty 
and staff typically respond by implementing summative assessments. The assessment results are 
reviewed by multiple audiences and used to make judgments about the quality or worth compared 
to previously defined standards for performance (Palumbo and Banta, 1999). During the 1990s, 
the attention of regional and professional accrediting associations turned to assessment of student 
learning. They require their members to have written assessment plans in place that can be 
periodically reviewed by accreditation teams. A focus on the data ramifications of competency- 
based initiatives, especially reliability and validity of competencies, can provide a basis for 
assessing student learning in line with expectations for continued accreditation. 

A central task for the Working Group and consultants, then, was to create a protocol for the case 
studies that would examine whether the concepts of reliability and validity were a concern. This 
protocol would also examine how, and if, these concepts were addressed in an operational and 
conceptual sense. The extent to which issues of reliability and validity are addressed affects 
transportability and plays a central role in shaping the data ramifications for a given initiative. 

The findings from these case studies will most likely be disappointing to those readers who wish 
to plumb the depths of data ramifications of competency-based initiatives. When they were 
examined, disappointingly few sites were actively dealing with issues of reliability and validity. 
Instead, as might be expected in some cases of relatively new endeavors, issues of planning, 
process, and internal acceptance for competencies took considerable time, energy, and resources 
that might otherwise be devoted to tackle transportability. One might expect that these case study 
sites may be unaware of other initiatives explored in this study and are connected only through 
this report. They do not share a common agenda for determining reliability and validity. 
Nonetheless, the reader can gain a sense of why data ramifications are addressed in some settings 
and not in others by reading the individual case studies. 
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3. METHODOLOGY FOR THE CASE STUDIES 



Multiple case studies that demonstrate exemplary competency- 
based initiatives are selected for in-depth study to identify their 
strong practices. 

This section presents a brief overview of the research methodology used to address the project’s 
goals. Initial discussions of the Working Group were guided by the creation of an annotated 
bibliography of selected resources about competency-based initiatives. This bibliography 
informed the selection of specific cases that fit into and addressed at least one major component 
of the framework. This framework (as outlined in section 2) provided an organizing structure to 
classify learning experiences prior to postsecondary education, during postsecondary education, 
and then exiting from postsecondary education or graduation as well as overall institutional 
effectiveness. An interview protocol was developed by consultants and guided all formal 
meetings with selected participants at each particular site. All data gathered from this project 
were analyzed to produce the project’s major findings. The major themes and important 
principles, entitled “strong practices” in this report, were found to cut across several cases as 
outlined in section 4. 



Annotated Bibliography 

Project consultants prepared an annotated bibliography (section 6) that provides a sample of 
competency-based initiatives that are encompassed by the classification framework developed by 
the Working Group. The bibliography is neither exhaustive nor comprehensive; rather, it was 
designed to provide illustrations of work completed on the points of transitions reflected in the 
classification framework. It was also intended to capture samples of how competencies are 
defined and measured. 

This bibliography provided insights about particular issues associated with the development, 
implementation, and assessment of competencies. It also provided directions for the 
identification and selection of several strong case studies. 



Selection of Specific Case Study Sites 

The Working Group identified potential case study sites that address at least one area of the 
framework for classifying competency-based initiatives. In addition, they wanted to examine 
institutions or corporate sites that (1) introduced students/employees to the competencies to be 
developed, (2) provided multiple opportunities for the competencies to be learned, and (3) 
assessed students/employees throughout their learning experiences. As the Working Group 
considered potential sites, they decided it would be best to select cases that have not been studied 
in depth by other researchers. For example, the Working Group decided that Alvemo College is 
well known for its high-quality competency -based work, but they believed that it has been studied 
extensively and is well documented elsewhere in the literature. The Working Group ultimately 
selected institutions (community colleges, universities, and a corporation) that had either strong 
competency-based initiatives already in place or were developing ambitious programs that could 
influence postsecondary education. The sites listed in exhibit 1 were part of this study and 
address at least one key transition point related with postsecondary education and competency- 
based programs. 
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These cases represent two very different ends of the continuum in terms of their own 
development and levels of experience. The institutions in one group (e.g., King’s College, 
Northwest Missouri State University, and Ford Motor Company) typically have more than 10 
years of experience with competency-based initiatives. Therefore, the development, 
implementation, and assessment of competencies have been integrated across their entire 
educational system. This means that competencies typically cut across all of their programs. For 
most institutions in this group, there are some preliminary findings about data ramifications, 
including reliability and validity. 

The other set of cases represents organizations (e.g., Western Governors University, Colorado 
Incumbent Worker Project, Hagerstown Community College — The National Science Foundation- 
funded College Consortium, and Proficiency-based Admission Standards System in Oregon) that 
are in the early stages of development and planning. They have not reached full implementation 
that includes assessments of student learning. These particular cases comprise groups of 
institutions working together toward major reforms that may be more time-consuming than those 
individual institutions with a long history of reform. Since these particular cases are in the early 
stages of development work on their competency-based educational initiatives, there is very 
limited information about data ramifications, and in some situations, there is no information 
available at the time of this work that addresses reliability and validity issues. The main focus of 
these partnerships is on their plans. The Working Group strongly believed this second set of 
cases was very important to examine at this time since they are ambitious and have great potential 
to influence postsecondary education. 



Interview Protocols 

The consultants developed an interview protocol that was evaluated and approved by the 
Working Group. The instrument served as a guide for all interviews. The protocol focused 
specifically on the areas outlined below, and for each specific topic, additional probing questions 
were pursued (see appendix I). 

• Explaining why a competency-based model was selected; 

• Describing how the competencies are used; 

• Outlining how the competencies were developed; 

• Identifying who provided leadership for the development of these competencies; 

• Addressing how reliability and validity of competencies are ensured; 

• Identifying how competencies are assessed or measured; 

• Highlighting challenges and successes in development, implementation, and 
assessment of competencies; 

• Determining if the competencies are transportable; and 

• Outlining important advice for other college and university faculty, researchers, 
and administrators who may be interested in implementing a competency-based 
model. 

The Working Group particularly wanted an emphasis on data and their ramifications. These 
points were gleaned from responses to the questions regarding the process of establishing 
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competencies and their assessment. Some interview questions focused upon the reliability and 
validity of competencies, as well as the assessment methods and results. 



Interview Participants 

For each case study, an institutional liaison (who provided leadership at the organization) 
identified the appropriate individuals to interview. For most cases, the consultants visited each 
site and interviewed 5 to 18 individuals. The participants frequently were faculty representing 
different disciplines and key administrators (including directors of assessment). These 
individuals often participated in the development, implementation, and assessment of the 
competencies. Most interviews lasted about 45 to 60 minutes. 

The Working Group consultants also collected relevant documents and materials that provided 
additional insights about the specific competency-based initiative. These materials often included 
college catalogues, strategic or assessment plans, and selected assessment instruments. 



Analysis 

The Working Group consultants analyzed the results for each question by identifying the major 
themes across the participants’ responses to questions within a single case. Individual case 
studies were then created and are abstracted in section 7; complete case study reports are 
provided in section 8. 

The Working Group consultant and members also analyzed the results by examining common 
themes across the case studies. Although a wide variety of initiatives was studied, there were 
frequently common issues. In addition, certain case studies portrayed strong practices in the 
development, implementation, and assessment of competencies. For this reason, the next section 
focuses on a cross-site analysis and articulates the most effective practices found among the sites. 
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4. PRINCIPLES OF STRONG PRACTICE 



Strong competency-based initiatives produce meaningful 
assessment results that are used in making critical decisions about 
ways to improve student learning. 

This section presents a cross-site analysis to facilitate a better understanding of common, strong 
practices. These strong practices are discussed in this section so that faculty and administrators 
can decide how these strategies can be adapted at their own college or university. These practices 
also suggest key considerations and processes that should be reflected upon as groups of faculty, 
administrators, or corporate leaders decide whether to implement competency-based models 
within their own organizations. 

Strong practices are clustered together under four main categories: 

• Planning for competency-based education initiatives; 

• Selecting assessment methods; 

• Creating and ensuring that learning experiences lead to competencies; and 

• Reviewing assessment results to identify changes needed to strengthen student 
learning. 



Most faculty and staff begin by identifying and defining important competencies for their own 
students. During this stage, they usually seek feedback from their stakeholders. Once this work 
is completed, faculty and staff review and select assessment methods or measures that are linked 
with the specific competencies that have been defined. Strong, multiple assessments provide 
faculty, staff, and administrators with meaningful information about whether students are 
mastering the important competencies and what levels of performance they have achieved. 
Faculty also strive to embed these competencies within courses, programs, and across the 
institution’s academic programs. Often faculty and staff use formal documentation methods 
(common to all programs) that outline the specific courses where students will learn about these 
competencies and how they will be addressed in terms of the design of learning experiences. 
Finally, once faculty and staff have completed initial assessments of student learning, they can 
begin to evaluate the data itself, whether there are certain areas of student learning that need to be 
improved, and where students demonstrate strong skills. 

Each principle derived from the case studies will be discussed, and appropriate references will be 
made only to institutions that were part of this research. One limitation of this approach is that 
only the particular case studies (examined in this research) that demonstrate strong practices will 
be referenced. There are other colleges and universities that already adhere to these practices 
(such as Alvemo College), but since they were not part of this research study, they are not cited. 
The strong practices presented in this report were derived solely from the institutions and entities 
selected by the Working Group. Another limitation is that institutions have substantially varying 
levels of experience with competencies based upon the number of years that they have been 
working toward their own implementation and assessment. Some institutions have a long history 
of work (more than 10 years) with competency-based curricula, while other sites are in the early 
stages of developing and planning for their initiatives. 
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The accumulation of all of these principles may seem overwhelming to faculty and administrators 
who are wondering whether to create a competency-based system. However, readers are advised 
to remember that institutions with the strongest practices have been working at this for many, 
years. Perhaps one of the greatest overriding lessons learned is to provide sufficient amounts of 
time and guidance to help faculty develop, implement, and evaluate their competency-based 
educational initiative. 



Planning for a Competency-Based Educational Initiative 

When the case studies are examined as a group, important strategies emerge that help sustain the 
initiative over time. Academic leaders, faculty, and staff follow these key principles (outlined in 
this section) in the planning process. All principles listed below are equally important, and the 
numbering does not indicate any type of rank ordering. 



The Planning Process: Strong Principle for Academic Leaders 



Principle # 1 : A senior academic administrator becomes the public 
advocate , leader, and facilitator for creating an institutional culture 
that is open to change, willing to take risks, and fosters innovations by 
providing real incentives for participants. 

■ Encourage faculty and staff to conceptualize and implement 
changes in an incremental manner; 

■ Provide sufficient amount of time and support to enable 
participants to fully reflect upon potential competencies, conduct 
formal research to identify the most important competencies, and 
reach consensus; 

■ Give faculty and staff sufficient training and development 
opportunities to learn how to identify and define competencies; 
and 

■ Analyze realistically budget needs and insure adequate resources 
are provided to develop competency-based initiatives. 



At most sites in this study, there was a key institutional leader who was perceived as the major 
visionary guiding faculty to consider how to develop a competency-based curriculum. Often a 
senior academic administrator met regularly with faculty leaders and relevant committees in an 
effort to structure how decisions would be made about transforming the curriculum into a 
competency-based model. Faculty and staff who participated fully in this change process often 
viewed their senior administrator as an individual who possessed the important leadership skills 
to build an environment of trust across the campus. This academic leader fostered ongoing and 
open communications with all relevant stakeholder groups. Faculty, staff, and administrators 
typically worked as partners in a collaborative process. Since these leaders placed an emphasis 
on a participatory process, most faculty felt ownership for the curriculum change and fully 
endorsed it. 
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The senior leader was fully committed to change since he or she provided real incentives to 
encourage innovation and support for the participants. Incentives for participation included 
effective, meaningful strategies to address the demands upon faculty time. Some professors were 
released from teaching a course in their regular load or they were paid stipends to participate and 
work on their initiatives in the summer. They also could receive support from an undergraduate 
or graduate student who was paid to work with them. Additional incentives included resources to 
attend professional conferences or to purchase necessary equipment. However, for faculty, the 
most important issue was their time. Therefore, the most meaningful incentives seemed to be 
those strategies that helped them to work effectively within their busy schedules and manage 
multiple priorities within reasonable amounts of time. 

These incentives typically cost money and require an academic leader to develop a realistic 
budget that will cover important expenses. In addition to incentives, there will be more start-up 
costs that an academic leader will need to analyze and account for in order to build a strong 
competency-based initiative. 

If change is introduced quickly and expected to affect the entire system simultaneously, then 
there is an increased likelihood that the initiative may face major challenges and not be sustained 
over time. A senior academic leader often encouraged incremental steps over time that 
eventually led to more comprehensive change across the whole system. For example, at King’s 
College, faculty and staff endorsed building a competency-based curriculum. Faculty identified 
the important competencies that should be a foundation for all programs. However, they decided 
to begin with the articulation of critical thinking competencies across the curriculum before 
proceeding to define and expect other important skills to be embedded across learning 
experiences (including both general education and all academic major programs). Such a strategy 
increases the likelihood that the initiative can be sustained over time. 

Large changes were broken into smaller, more manageable steps that could occur and build over 
time. In addition, faculty and staff were encouraged to pilot test their innovations so that they 
could determine what was working and what could be improved. Faculty seemed to feel less 
threatened and more confident by knowing that they could try new ideas that might not succeed 
without serious consequences. The academic leaders typically believed that even if a pilot did not 
fully succeed, at least faculty and staff had attempted new ideas and could learn what should be 
changed or revised the next time they implemented their ideas. Finally, academic leaders 
encouraged faculty to take sufficient amounts of time to develop their ideas. For example, the 
consortium of community colleges spearheaded by the Johns Hopkins University spent a full year 
working with faculty and employers to identify and define the important competencies for then- 
programs and courses. Participants in the Western Governors University also found that it took a 
full year to identify and define competencies. Although it may be tempting to rush forward with 
this process, academic leaders found that adequate time is very important in building a stronger 
curriculum. Spending a full year on defining and identifying competencies was typical since these 
are frequently new concepts for faculty and staff who have not investigated the needs of 
employers or other relevant stakeholder groups. 

Another critical and necessary element was the creation of strong faculty development programs 
so that participants can learn how to identify and define competencies. Faculty and staff are often 
open to new ideas but have not learned through their graduate studies about how to define 
specific competencies that can actually be assessed. Faculty development initiatives usually rely 
on a combination of internal experts and external consultants who work with faculty on designing 
their course and program plans. Through a series of planned workshops often convened during 
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the summer, faculty and staff get time to focus their attention on redesigning their courses. Such 
formal workshops allow faculty and staff to experiment with new ideas, then they receive 
informed advice from experts about how they might strengthen their definitions of competencies. 
These workshops are usually offered on a regular, consistent basis with followup discussions and 
meetings rather than a single event. Followup meetings and ongoing workshops are important to 
reinforce the ideas of faculty and staff and allow for meaningful revisions over time. 



The Planning Process: Strong Principles for Faculty and Staff 



Principle # 2 : The appropriate stakeholders fully participate in 
identifying , defining, and reaching a consensus about important 
competencies. 

■ Review existing, relevant information about important 
competencies and either adopt or tailor own competencies based 
upon multiple resources; 

■ Develop formal process for identifying and defining competencies 
that is credible to key stakeholders (students, faculty, employers, 
and policymakers); 

■ Set standards or criteria that indicate the required levels of 
performance; and 

■ Share relevant information about specific definitions of 
competencies, how they were identified, and the standards 
associated with adequate mastery with appropriate stakeholders. 

Principle #3: Competencies are defined at a sufficient level of 
specificity that they can be assessed. 

■ Define multiple competencies that capture progressively more 
advanced knowledge and skills; and 

■ Target both content (knowledge) as well as generic and process 
skills (such as writing or oral communication). 

Principle # 4 : Competencies are clearly defined, understood , and 
accepted by relevant stakeholders. 



Across several of the case studies, faculty and staff began to re-evaluate their own curriculum 
because they were dissatisfied with the existing version. Faculty often believed that some 
students were not well prepared upon graduation and lacked the necessary skills and abilities to 
perform at a high level in the workplace and in society. Faculty also stressed that their 
curriculum lacked coherence and that students would take courses from a fragmented, piecemeal 
set of learning experiences. Without meaningful coherence that would unite courses and 
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programs with some common goals, faculty thought that students were completing an education 
based upon individual learning experiences that did not necessarily help students reach their full 
potential. Typically in the older versions of their curriculum, faculty had not identified the 
essential, specific competencies that students should achieve, nor did they build coherence across 
learning experiences. 

When dissatisfaction reached a peak with the old curriculum, then faculty began to review their 
curriculum formally and moved toward identifying and articulating relevant and current 
competencies that students would need to be effective in the real world. Initially, faculty often 
reviewed multiple, external sources of information about relevant competencies before they 
began their own decision-making process. Sometimes faculty were influenced by professional 
accreditation requirements or their own professional organizations (such as the American 
Accounting Association). Such organizations typically articulated extensive lists of relevant 
competencies that faculty groups would review and consider. The Secretary’s Commission on 
Achieving Necessary Skills 3 (SCANS, 1992) was another resource that community college 
participants frequently considered, as were occupational skills standards and other industry 
specifications about important competencies. 

Once faculty have identified and reviewed relevant, potential competencies, they must determine 
the best strategy for getting formal feedback from their own constituencies. Potential participants 
include the faculty, alumni, current undergraduate students, and employers who hire the 
program’s or institution’s graduates. These formal strategies to identify the relevant 
competencies are typically research-based and require a systematic analysis of results in order for 
the information to be meaningful, useful, and valid. While it may be tempting to make decisions 
about competencies by using intuition — one’s own experience, imitation, or random discussion — 
faculty in these case studies tended to interview or to survey relevant stakeholder groups. Some 
faculty leaders used both strategies. The Johns Hopkins University-led consortium of community 
colleges initially needed to identify important competencies for their manufacturing majors. 
They used several sets of external resources including SCANS, an existing DACUM (Developing 
a Curriculum in Manufacturing) report, and the skills-setting efforts of participating industry 
representatives (Packer and Mathias, 1995). Then members of this consortium spent a full year 
using a modified DACUM process to formally gather feedback from employers (in the relevant 
industries) and faculty. Drawing upon the multiple resources identified above, the leaders began 
with a list of competencies and asked faculty and employers to identify the most important areas 
to focus on within the curriculum. In a similar manner, the president of Northwest Missouri State 
University conducted a Delphi study to identify the essential competencies that undergraduates 
should master to be effective in the workplace. The results from this research became the 
foundation for the entire curriculum and defined key quality indicators (such as communications, 
problem solving, and self-directed learning). 

Faculty in several of these case studies began with identifying a general goal, but they also 
proceeded to define specific competencies that could be assessed. For example, faculty might 
state that they want all undergraduates to “develop effective critical thinking skills.” While this 
goal is very important, it is open to different interpretations. Faculty might have different views 
on what constitutes “effective” critical thinking if they were asked to define it based upon their 
own individual experiences. Prior research has identified the different dimensions associated 
with critical thinking, including evaluation, analysis, inference, interpretation, presentation of 
arguments, and related dispositions (being open-minded or flexible) (Jones, 1994; Facione, 1990). 



3 This Commission was appointed by the Secretary of Labor. 
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The accounting faculty at King’s College wanted their majors to develop strong critical thinking 
skills. They felt their students should be able to apply critical thinking skills in preparing and 
analyzing financial statements, preparing budgets, making managerial decisions, preparing audit 
programs, and writing audit reports. They established specific critical thinking competency 
statements for specific accounting courses, indicated assignments that would help students master 
these skills, and articulated specific assessment criteria. One abbreviated example from an 
advanced accounting course is shown in exhibit 2, and additional illustrations are provided in the 
entire case study reported in section 8. 

Exhibit 2, — Abbreviated example of a competency statement* * 



Competency description Strategy 



Assessment criteria 



The student will be able to: 



Assignments: 



Bring together two separate 
financial statements of 
corporations under common 
ownership utilizing the pooling 
of interests and purchase methods. 

Understand the accounting 
alternatives and history of 
goodwill resulting from 
business combinations. 



3. Understand the effect of 
globalization of business 
upon the reporting of 
business transactions, including 
foreign currency transactions. 



Reading and reporting 
on articles from financial 
publications. 



Solving problems related 
to various subjects on both 
a manual basis and spread- 
sheet software designed for 
consolidated financial 
statements, foreign 
currency remeasurements 
and translations, and 
partnership accounting. 

Answering essay 
questions and solving 
problems. 



The student correctly 
selects, compiles, and 
analyzes data needed 
to prepare meaningful 
financial reports. 

The student prepares 
reports according to 
generally accepted 
accounting principles. 



The student identifies 
alternative perspectives 
on issues presented in 
various segments of 
the course. 

The student evaluates 
the merits and drawbacks 
of alternative positions, 
which can be chosen 
from competing 
alternatives in the 
various segments of 
class. 



4. Account for various types 
of partnerships and the 
unique characteristics 
between partners and the 

partnership. 

*Reprinted with permission of Donald Farmer, King’s College, Wilkes-Barre, PA. 



The student shall use 
logical reasoning 
throughout the course 
in understanding 
competing alternatives. 
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Faculty typically articulate competencies that cut across both content areas as well as more 
generic process skills. Nearly equal weight or attention is typically placed upon both content and 
the generic skills so students are getting opportunities to develop them across courses. Faculty 
expect students to master competencies at a basic or foundation level that stress acquiring basic 
information and understanding it. However, even beginning in the freshman year, faculty expect 
students to master more sophisticated competencies that require advanced reasoning or problem- 
solving abilities. For example, at King’s College, faculty identified a series of general liberal arts 
learning skills (effective writing, effective oral communication, library and information 
technology, computer competencies, critical thinking, and quantitative reasoning) that were 
embedded within all academic programs and within individual courses. At. Ford Motor 
Company, about two-thirds of their competencies were centered on the generic process skills 
(including teamwork) with a particular focus on innovation, risk-taking, and creativity. In 
addition, across these process skill competencies, there were specific statements about diversity 
and cultural sensitivity. The remaining one-third of their competencies directly addressed the 
business knowledge that their employees need to be effective in their positions. 

Why was it important to define specific competencies? One main reason was that specific 
articulations of competencies informed and guided subsequent assessments at the course, 
program, and institutional levels. Secondly, specific competencies allowed faculty and students 
across campus (as well as other stakeholders such as employers or policymakers) to have a 
common understanding about the specific skills and knowledge that undergraduates should 
master as a result of their learning experiences. Assuming that faculty used a formal process to 
get feedback about what the competencies should be, stakeholders were more likely to accept and 
value them. Third, specific competencies provided directions for designing learning experiences 
and assignments that helped students gain practice at using and applying these competencies in 
different contexts. 



Selecting Assessment Methods for the Competency-Based Educational Initiative 

In these case studies, once faculty and staff had identified and defined the important 
competencies, they typically decided which particular assessment methods seemed to be the 
strongest in terms of measuring these competencies. 



Selecting Assessment Methods: Strong Principles for Academic Leaders 



Principle #5: Multiple assessments of competencies provide useful and 
meaningful information that is relevant to decision-making or policy- 
development context. 

Principle # 6 : Faculty and staff fully participate in reviewing and 
making decisions about the strongest assessment instruments that will 
measure their specific competencies. 

■ Consider student perspectives in assessment decisions; and 

■ Provide formal training for faculty and staff to learn how to assess 
competencies and make informed decisions about the strongest 
instruments. 
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Faculty and staff usually benefited significantly from formal training in reviewing and selecting 
the strongest instruments. In addition, if they decided to create their own instruments, they 
needed help in designing them. Internal experts, especially social science faculty, institutional 
researchers, and assessment staff, provided guidance to faculty and critiqued their instruments 
mainly to offer suggestions for improvement. Staff members with strong expertise in technology 
offered recommendations about how assessments may be designed using current technological 
capabilities that already existed on campus. Faculty development was a serious endeavor at 
King’s College, where professors were encouraged to participate in an ongoing series of seminars 
during the summer or a particular semester. The goal was to help faculty gain self-confidence 
and to be prepared professionally to implement their new assessment ideas. Group training in 
computer literacy, critical thinking, or writing across the curriculum were necessary for faculty to 
become responsible for the total education of their students (Farmer, 1988). 

Faculty and staff participation in selecting the strongest and most appropriate instruments to 
assess student learning was crucial. Faculty and staff were more likely to trust and value 
assessment results if they were consulted during the instrument review process and before 
decisions were made to use them. Assessments were important since students were asked to 
demonstrate what they knew or could do with their knowledge. Faculty and staff tried to select 
assessments that would provide meaningful and useful information to inform key audiences about 
student learning. However, targeted audiences, such as policymakers, may value certain types of 
instruments more than other methods. For example, in Missouri the members of the state’s 
Higher Education Coordinating Board initially believed that commercially developed tests were 
the main instruments that provided meaningful data. This resulted in the Board initially 
providing funds only for colleges and universities that used such instruments in their assessment 
programs. Recently, the Board began to provide resources to institutions that want to develop 
local assessment methods. However, they still provide greater sums of money to colleges that use 
commercially developed tests. In reality, this type of belief among key policymakers can 
influence the decisions that local academic leaders make, especially given limited resources. 

Faculty and staff chose multiple measures to directly assess certain competencies.. For example, 
faculty at Northwest Missouri State University used the Academic Profile to assess writing skills, 
college-level reading, and critical thinking of undergraduates. However, they also developed 
their own assessment instrument whereby students were asked to read multiple documents about 
a real issue or problem and then state their positions or justify the best solution to a problem in 
writing. Faculty designed this approach to determine if students were mastering the necessary 
writing competencies. The writing faculty worked together to develop and implement scoring 
rubrics to assess student work. Undergraduates were required to take this writing assessment at 
the end of their second writing course. Students were typically given a series of short, related 
readings (often from newspapers or magazines) about a controversial topic and then asked to 
respond to specific prompts. 

Faculty also frequently used a combination of course-embedded assessments and locally 
developed assessments that were administered outside class. Most faculty believed that students 
were more highly motivated to take the assessments seriously when they were embedded within 
existing courses. However, such assessments that were designed by an individual faculty 
member may only assess his or her own particular expectations for student learning. When 
faculty collaborated together to articulate common expectations and develop some common 
assessments, they can glean useful information about student performance. Faculty at King’s 
College in each major program area developed sophomore-junior projects to assess students’ 
abilities to transfer thinking and communication skills to a selected question or project relating to 
their academic field of specialization. This assessment served as a screening device to identify 
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students who needed follow-up support services that responded to their deficiencies. 
Furthermore, a senior level, integrated assessment helped faculty in students’ majors to make 
judgments concerning students’ readiness to function at an effective level for meeting the 
expectations of faculty and employers. 



Dealing with Data Ramifications: Strong Principle for Faculty and Staff 



Principle #7: The precision, reliability, validity, credibility, and costs 
are all considered and examined in making selections about the best 
commercially developed assessments and/or locally developed 
approaches. 



Faculty and staff at Northwest Missouri State University used both commercially developed tests, 
such as the Academic Profile, and locally developed approaches. The Academic Profile has a 
record of reliability and validity since a commercial organization has used experts to assess these 
dimensions over a long period of time. However, such instruments may not always capture all of 
the specific competencies that faculty expect students to master. Therefore, at Northwest 
Missouri State University, the faculty also created their own scoring rubrics to assess students’ 
writing competencies. They then spent considerable time training all raters. Initially, all raters 
reviewed the same two to three different samples of student work and then determined how close 
their ratings were to each other’s scores. When there were major disagreements in ratings, they 
discussed their reasons for assessing student work in a certain way. These discussions and 
reviews of rationales for their individual judgments helped to strengthen the consistency of their 
ratings. Faculty viewed this process as inordinately time consuming, but all faculty raters were 
paid by the hour (as overload pay) for their work in reading student essays. The formal training 
that faculty developed for all reviewers and raters of student work helped to promote the 
reliability of these assessments. Both reliability and validity were enhanced by the use of clear, 
articulate scoring rubrics (Palumbo and Banta, 1999). In addition, the use of a real-world 
problem presented from the context of several different sources of information helped students to 
examine an issue and justify their own solutions. 



Creating and Ensuring Learning Experiences Lead to the Mastery of Competencies 

In the majority of these case studies, academic leaders, faculty, and staff intentionally planned 
and explicitly linked their competencies across several levels. Competencies were embedded 
within individual courses. They were also aligned with the overall goals of academic programs 
and with general education. These linkages were crucial and fostered greater coherence in the 
curriculum. If students were to reach the intended competencies, they needed multiple 
opportunities across the curriculum to learn, practice, and master these skills. In addition, these 
clear linkages also should lead to more meaningful data. 
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Linking Competencies Across the Institution: Strong Principles for Academic 

Leaders, Faculty, and Staff 



Principle # 8 : The competency-based educational initiative is embedded 
within a larger institutional planning process. 

■ Directly link and align competencies with the goals of courses, 
academic programs or majors, and/or general education; and 

■ Demonstrate these linkages by providing documentation that 
clearly illustrates each linkage. 

Principle #9: The assessments of competencies are directly linked with 
the goals of the learning experience. For example , assessment 
processes are designed to measure the intended outcomes of the course 
or major. 



At King’s College, a set of six competencies was embedded within individual courses in both 
general education and across all academic programs. Faculty identified the specific courses 
where students will master the competencies beginning in the freshman year and continuing 
through the senior year. The official documentation that all program faculty completed was 
referred to as “competency growth plans.” Faculty initially identified the general competency 
and then provided a more specific description of each competency by outlining an average of five 
to seven additional statements. Next, the strategy for getting students to work on these 
competencies was highlighted. These strategies were typically the specific types of assignments 
that students completed in their courses, such as argumentative essays. Finally, faculty identified 
specific assessment criteria that guided the actual course-embedded assessments of student work. 
Faculty at Northwest Missouri State University developed similar plans linking their “key quality 
indicators” across both general education and all academic programs. Faculty were also required 
to document their work and explicit linkages by integrating these features into their strategic 
plans. 

When faculty fully developed and articulated these important linkages, they created strong 
curriculum plans that address the following questions; Where in the curriculum did students learn 
and practice skills such as writing, critical thinking, speaking, and teamwork? What teaching 
strategies and assignments were given to students to help them develop these skills? What 
assessments were selected to determine if students were mastering the competencies? What types 
of skills (such as problem solving or writing) should be required across courses in general 
education and all majors? When faculty reflected upon, discussed, and answered these questions, 
they clearly demonstrated their plans through formal documentation that was completed and 
shared with relevant stakeholder groups. 



Reviewing Assessment Results to Identify Changes to Strengthen Student Learning 

Once faculty and staff gathered assessment data, then they analyzed and interpreted the results. 
The main goal was to report the assessment findings in a clear and meaningful way that would be 
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understood by diverse stakeholder groups. The greatest challenge was to use the assessment 
results to make important decisions to strengthen student learning. 



Reviewing Assessment Results: Strong Principles for Academic Leaders, Faculty, 
and Staff 



Principle #10: The assessment results are used in making critical 
decisions about strategies to improve student learning or allocation of 
resources. 

■ Evaluate assessment processes and methods periodically to 
identify appropriate revisions; and 

■ Generate assessment results that are reliable and can be replicated 
(e.g., the same results are achieved when assessments are repeated 
under the same circumstances). 

Principle #11: The assessment results are clear and reported in a 
meaningful way so that all relevant stakeholders fully understand the 
results. 

■ Share the results on a regular basis with appropriate stakeholders; 
and 

■ Teach and advise target audiences to effectively interpret and 
apply assessment results to their decision-making and policy- 
development responsibilities. 

Principle #12: The institution experiments with new ways to document 
students ' mastery of competencies that supplement the traditional 
transcript. 

■ Document student learning in new ways that are meaningful and 
clearly understood by key stakeholders including students and 
employers. 



As faculty and staff implemented their competency-based educational initiatives, they often 
experimented with new ways to report and share student learning. Some employers were 
dissatisfied with traditional transcripts. The National Science Foundation (NSF)-funded 
consortium at Hagerstown Community College explored the best way to create a career transcript 
by seeking feedback from employers about what type of information was useful and meaningful 
to them. This career transcript is still under development. However, it most likely will 
demonstrate the levels of student mastery of specific competencies (i.e., in problem solving or 
teamwork). One major policy issue associated with the career transcript is determining who will 
have control over the information that gets entered into this official document that will be shared 
with college administrators and employers. The NSF consortium let students decide what 



assessment results they wanted to have placed on their career transcripts. If students performed 
poorly on a certain competency assessment, they could decide to suppress this information by not 
reporting it on the career transcript. 

As faculty and staff reviewed their assessment results, the entire set of processes was typically 
evaluated over time and subject to change. If they found that certain processes were not working 
effectively, they had the opportunity to revise their work. In the NSF-funded consortium work, 
they found that managers or recruiters from different companies did not understand the 
assessment results that were reported on the career transcripts. Therefore, this group worked on a 
second iteration and continued to seek feedback from employers who typically hired their 
graduates. The main goal was to express the competency-based assessment results in a 
meaningful and clear manner so employers could fully understand the career transcript. 

If competencies were clear, meaningful, and fully understood by relevant stakeholders, there was 
greater likelihood that linkages across systems could be built. However, equally important were 
the assessment results and whether there were high levels of reliability and validity. If the 
stakeholders were satisfied with the levels of reliability and validity, it was more likely that such 
assessments were used to gauge student mastery. If students reached sufficient levels of 
performance as evidenced by their results on competency-based assessments, they were likely to 
get the appropriate courses waived or to receive credit based upon their assessment results. 

The strongest potential for this type of competency-based system across secondary schools, 
community colleges, and universities was evident in the Proficiency-based Admission Standards 
System developed in Oregon. This system required students to demonstrate proficiencies in six 
subject areas: math, English, social science, science, visual and performing arts, and second 
languages. These proficiencies were being phased in as admissions requirements for colleges and 
universities in Oregon over a 4-year period. A benefit of this type of system is that high school 
students know precisely what is expected of them. In Missouri, the Higher Education 
Coordinating Board’s mandate that all institutions follow the same set of goals for general 
education also helped to create smooth transfers for students who wanted to enter another 
institution within the state. 



O 

ERIC 



30 



46 



5. CONCLUDING OBSERVATIONS 



Future work should continue to examine issues of transportability 
of learning experiences across academic programs and 
institutions by focusing on how reliability and validity are 

addressed. 

The case study institutions and entities discussed in this report illustrate strong practices that 
ensure the effective use of competencies. While some innovations may begin with support from 
external grants by funding agencies, these initiatives are difficult to maintain without committed 
campus leadership and a solid sense of ownership from faculty and staff. If academic leaders 
want changes to have a real impact on student learning, they must enlist the services of broad 
groups of faculty, staff, and administrators — rather than individual pockets of faculty — to work 
on the initiatives, gradually implementing change across the curriculum with the goal of 
strengthening students’ skills and competencies. 

As work proceeds on competency-based educational initiatives, faculty and staff will probably 
fmd that two of the most challenging tasks are identifying and implementing strong assessment 
methods. They must then accomplish the equally challenging task, particularly critical for 
systems of higher education, of ensuring that the assessment methods are reliable and valid. 
Even institutions with a long history of defining and assessing competencies may find it difficult 
to monitor, maintain, and document the reliability and validity of their assessment work because 
faculty and staff may not fully understand these two technical concepts. The case studies 
exemplified in this report can be useful in guiding internal institutional discussions about the 
groundwork necessary to address this critical area. 

Faculty and staff who use commercially developed assessment methods produced by national 
testing companies frequently rely on these organizations to document that their testing methods 
are reliable and valid. Some dissatisfied faculty do not believe that commercial tests accurately 
or meaningfully measure whether students have achieved the education goals specific to the 
academic program or institution. In addition, they fmd that these instruments do not provide 
much direction nor guidance about where and how to improve student learning. For example, 
students may perform poorly on a critical thinking test, but such results do not tell faculty exactly 
which dimensions of critical thinking need to be improved. 

Given this dissatisfaction, an increasing number of faculty are developing their own assessments, 
particularly in writing. They are addressing reliability issues by training assessors extensively 
about how to score student work consistently over time. Although there are limited data about 
the actual reliability and validity of these assessments, faculty and staff do benefit from formal 
training in these areas and from seeking advice or help from their on-campus experts. Such 
experts include institutional researchers and assessment staff, as well as faculty members in the 
social sciences with experience in statistics or educational psychology. These experts can help 
faculty create strong, locally developed instruments and then track their reliability and validity 
over time. In future work, faculty can explore ways to design local assessments that are 
increasingly reliable and valid. 
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Some states, such as Missouri, are working toward a transportable system of postsecondary 
education so undergraduates can move between institutions more smoothly. It is unlikely that a 
national system, where competencies have a common meaning in a variety of settings, will be 
developed and accepted among institutions. However, academic leaders, faculty, and staff within 
individual states are finding that they can work collaboratively to describe, package, and record 
student learning in terms of specific competencies. Such initiatives will clearly demonstrate the 
specific skills and levels of learning that ultimately should provide meaningful information to 
employers who hire college students. 
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6. ANNOTATED BIBLIOGRAPHY 



The purpose of this bibliography is to be a reference for people interested in using competencies 
as an alternative to more traditional measures of student achievement in postsecondary education. 
As such, it is not meant to be exhaustive; rather, it is illustrative of the many works available 
about competencies — their definition, measurement, and use. “Competencies” as used in this 
bibliography refer to specific statements of student performance in terms of skills, knowledge, 
and abilities. Several questions guided the choice of articles, books, and monographs to be 
included. First and foremost, were competencies explicitly used? Were there explicit statements 
of attempts to grapple with student competence? How were those explicit competency statements 
used in the processes of the activity? How were competencies documented? How were 
competencies particularly useful in the transition from one activity to another? 

In taking this approach, certain bodies of literature were not included in order to keep the 
bibliography manageable — for instance, numerous volumes on performance indicators, 
performance funding, accountability, and assessment were omitted. The number of institutional 
examples was limited as well, in order to focus on the use of competencies in larger processes. 
The constraint of not giving too many individual institutional citations curtailed the number of 
practical examples given, because many institutions document narrowly and do not contain 
guidelines on how to adapt approaches to a variety of institutional settings. 

Many possibilities of specific occupational and industry-specific competency lists that have been 
compiled by various groups have not been included. For those interested, a few* suggested 
starting points are the National Skill Standards Board Web Site (www.nssb.org), which includes 
links to state skill standards sites; the Occupational Skill Standards Projects at 
www.ed.gov/offices/OVAE/OccSkills/index.html; the Secretary’s Commission on Achieving 
Necessary Skills site at Johns Hopkins University (www.scans.jhu.edu/default.htm); 0*NET, the 
Occupational Information Network, at www.doleta.gov/programs/onet/; and the National Center 
on Education and the Economy (www.ncee.org). 

This bibliography is composed of five sections. The first is a general introduction to the history 
and basics of competencies in postsecondary education. The second contains citations about 
entry into postsecondary education, including competency-based admissions and placement. The 
third section focuses on the use of competencies within postsecondary education, such as 
competency-base curricula, general education competencies, and transfer competencies both 
within and across providers. Exit from postsecondary education, including end-of-program 
competencies, those used for employment placement, and those used for admittance to graduate 
or professional schools, are found in the fourth section. Finally, the fifth section is about the use 
of competencies for overall institutional effectiveness, including program improvement, 
accountability, performance budgeting, and accreditation. 





35 



Part I. General Introduction to Competencies in Postsecondary Education 

This section is included for the novice who is in need of a general introduction to the use of 
competencies in postsecondary education. Included here are basic materials about the 
construction of competencies as well as the historical development of the use of competencies. 



Basics of Competencies 

Mager, R. F. (1997). Preparing Instructional Objectives: A Critical Tool in the Development of 
Effective Instruction (3 rd ed.). Atlanta, GA: The Center for Effective Performance, Inc. 

Mager is considered one of the leading experts on instructional objectives. This book is a 
straightforward, how-to book on constructing usable instructional objectives 
(competencies). The strength of his approach comes from its applicability to both 
education and corporate training situations. His book is written with a broad, accessible 
perspective. 

Parry, S. B. (1998). Just What is a Competency? (And Why Should You Care?) Training , 

(June): 58-64. 

This article gives a business- and training-oriented perspective about the important 
differences between competencies and traits or characteristics, skills, or abilities, as well 
as between competencies, styles, and values. For Parry, competencies are generic, 
limited to 10 to 14 per company, and how individuals act them out is dependent largely 
on personality (“style/values”). Assessment techniques advocated include 360-degree 
feedback, role playing in a controlled laboratory situation, and use of interactive media. 

Walvoord, B. E., and Anderson, V. J. (1998). Effective Grading: A Tool for Learning and 
Assessment . San Francisco: Jossey-Bass Publishers. 

Walvoord and Anderson’s book is about making explicit the criteria used in grading. Of 
particular interest are Chapter 5, “Establishing Criteria and Standards for Grading,” and 
Appendix C, “Examples of Primary Trait-Based Scales Developed by Faculty.” These 
are useful because of the explicitness necessary when writing competency statements and 
when agreeing on the criteria used to evaluate competence. Although these pages focus 
on a specific technique, Primary Trait Analysis, the principles and processes are useful 
when working with competencies. 



Historical Background of Competencies 

Doll, W. E., Jr. (1984). Developing Competence. In Competence: Inquiries into its Meaning 
and Acquisition in Educational Settings , edited by E. C. Short, 123-138. Lanham, MD: 
University Press of America. 

Grant, G., and Associates. (1979). On Competence: A Critical Analysis of Competence-Based 
Reforms in Higher Education. San Francisco: Jossey-Bass Publishers. 

This volume is one of the few available that has as its subjects both competency and 
higher education; most of the other general books on competency-based education noted 
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in this bibliography are about elementary and secondary education. A definition that is of 
value is offered early in the book: “Competence-based education tends to be a form of 
education that derives a curriculum from an analysis of a prospective or actual role in 
modem society and that attempts to certify student progress on the basis of demonstrated 
performance in some or all aspects of that role. Theoretically, such demonstrations of 
competence are independent of time served in formal educational settings” (p. 6). Of 
particular note are the chapters by David Riesman, “Society’s Demands for 
Competence,” and Thomas Ewen, “Analyzing the Impact of Competence-Based 
Approaches on Liberal Education.” Riesman gives a historical summary of societal 
forces and why they resulted in competency-based education. Ewen grapples with the 
ever-present tension between the concepts of liberal education and competence. 

Klemp, G. O., Jr. (1979). Identifying, Measuring, and Integrating Competence. In Defining and 
Measuring Competence, edited by P. S. Pottinger and J. Goldsmith. New Directions for 
Experiential Learning, No. 3,41-52. San Francisco: Jossey-Bass Publishers. 

This chapter from a late 1970s monograph contains a useful discussion about the 
definition of competence, the interaction of competence and performance, and integration 
of competencies. Klemp also discusses responses to testing for competence illustrating 
the problems with traditional testing and suggests “intermediate criteria” for grading — 
efficiency, parsimony, thoroughness, and outcome effectiveness. He acknowledges that 
there is no one right answer: “A wide range of permissible responses helps prevent 
‘ceiling’ and ‘floor’ effects that restrict the useful range of competency measurement” (p. 
48). 

Urch, G. E. (1975). A Philosophical Perspective on Competency Based Education. In The 
Foundations of Competency Based Education, edited by R. T. Utz and L. D. Leonard, 30- 
47. Dubuque, IA: Kendall/Hunt Publishing Co. 

This chapter and the Doll entry above are for those who are interested in the 
philosophical background of competency-based education. They are from the 1970s but 
remain salient. Doll distinguishes “competence” and its relationship to “performance” 
based on Chomsky, Piaget, and Bruner. Urch cites critics, including Neill, Goodman, 
Rogers, and Illich, and then traces the “roots” of competency-based education back to 
Rationalism (Descartes and Spinoza), Empiricism (Bacon, Locke, Herbart, and 
Thorndike), Pragmatism (Dewey), and Behaviorism (Skinner and Gagne). 



Part II. Competencies for Entry into Postsecondary Education 

Citations in this section of the bibliography are about the use of competencies in the 
undergraduate admissions process. The other topic pertinent to this section is placement 
examination. However, very little has been written about the use of competencies in collegiate 
placement examination. Although some assessments and testing products are based on 
competencies (such as the Regents Examinations, New Standards, and ACT Work Keys), there is 
little documentation of their use in postsecondary education either (1) specifically as placement 
examinations, or (2) because of their focus on competencies. 
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Admissions 



Conley, D. T. (1996). Daddy, I’m Scared: A Prophetic Parable. Phi Delta Kappan, 78(4), 290- 
297; and Where’s Waldo? The Conspicuous Absence of Higher Education from School 
Reform and One State’s Response. Phi Delta Kappan, 78(4), 309-3 14. 

These two articles by Conley are about the Proficiency-based Admission Standards 
System (PASS) Project in Oregon. PASS is a high school graduation/college admission 
system based on proficiencies (competencies). The “prophetic parable” article is about 
the context PASS operates in, how they determined the proficiency standards, and how 
. he believes those standards will force higher education to change, particularly with regard 
to admission requirements. “Where’s Waldo?” focuses on how the PASS project fits into 
statewide education reform across the K-16 continuum. A balanced appraisal is given of 
opportunities and obstacles. 

(See also the PASS Web Site: http://www.ous.edu/pass/documents/archive). 

Conley, D. (1999). Statewide Strategies for Implementing Competency-Based Admissions 
Standards. [On-line], Available: www.sheeo.org/publicat/pub-kl6.htm 

Conley summarizes the results of a survey and case studies done by the State Higher 
Education Executive Officers organization regarding competency-based admissions. 
These systems are defined as “requiring] students to demonstrate their knowledge and 
skills in specified academic areas and at specified performance levels.” Eleven states 
replied affirmatively that they were considering or implementing competency-based 
admissions systems. Six states (CO, GA, MD, OH, and WA, WI) were chosen for 
additional study. The document includes a discussion of the policy context and reasons 
for considering a competency model; a short overview of the development, 
implementation, and assessment of competencies; what the results have been so far in 
these states, and what the continuing problems are in the conversion to such systems. 
Conley notes that whether these competency-based systems work “better” than traditional 
admissions systems is being closely watched. 

Roberts, R., and Robson, R. (1998). Electronic Transcripts in the Age of Standards. [On-line], 
Available: www.ous.edu/pass/documents/archive (click on documents, then down to 
archive). 

This is a short overview of how electronic transcripts are specified and will be 
constructed for use with the Oregon PASS (Proficiency-based Admission Standards 
System). They contain three levels of data. The first is “summative scores or binary 
decisions” regarding a student’s achievement. The second level will be “verifications” of 
these decisions and will consist of who made the decision, based on what, and when the 
decision was made. Finally, the third level is the actual work on which the student’s 
achievement level is based. 

Sills-Briegel, T., Fisk, C., and Dunlop, V. (1996). Graduation by Exhibition. Education 
Leadership, 54(4), 66-7 1 . 

This article is an example of how competencies are documented and assessed as a high 
school graduation requirement. This requirement — an individual thesis project — uses a 
committee of three overseers to work with each student. The committee comprises one 
instructor randomly assigned to the student and two people of the student’s choice (one 
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often comes from outside the high school, either a member of the community or from a 
local university). Levels of achievement are “not proficient,” “proficient,” and 
“distinguished performance.” Criteria used to evaluate the projects include personal 
responsibility, critical thinking, writing, public speaking, and multimedia presentation. 
No indication is given as to how graduates use these projects either as supporting 
evidence for college admission or for job attainment. 

University of Wisconsin System. (1998 Spring). Final Report: The University of Wisconsin 
System Competency-Based Admission Pilot Project. [On-line]. Available: 
www.uwsa.edu/acadaff/cba/report.htm 

In 1992, University of Wisconsin began studying competency-based admissions as an 
alternative to traditional admissions for those students attending high schools with 
nontraditional curricula. It is now an option available to students, although it is not 
meant to replace the existing admissions system. The final report is useful because it 
contains data comparing traditional admissions measures (ACT and grades) with the new 
competency-based admissions measures, as well as data on retention of students admitted 
using the competency-based approach. Measures of time put into the admissions 
decisions by university personnel and a sample admission reporting profile are also 
given. 



Part III. Competencies Within Postsecondary Education 

The purpose of this section is to highlight the use of competencies within postsecondary 
education. Competencies may be used in a variety of ways, including as curricular guides, as a 
means of validating experiential education and knowledge learned outside a formal educational 
setting, for specifying and easing articulation and transfer, and as feedback for students regarding 
their progress and learning expectations. 



Experiential Education 

Lamdin, L. (1992). Earn College Credit for What You Know. 2 nd ed. Chicago: Council for 
Adult and Experiential Learning. 

Whitaker, U. (1989). Assessing Learning: Standards, Principles, & Procedures. Philadelphia: 
Council for Adult and Experiential Learning. 

Although both books are written from the perspective of awarding credits (due primarily 
to the prevalent academic climate when they were first published), the .standards 
proposed and the information given are applicable to competency-based processes and 
projects. Whitaker outlines and explains the standards for Prior Learning Assessment as 
designed by the Council for Adult and Experiential Learning. Lamdin affirms the many 
nonacademic, nonclassroom methods of learning. 



General Education 

Alvemo College Faculty. (1994). Student Assessment-as-Learning at Alverno College. 3 rd ed. 
Milwaukee: Alvemo College. 



Farmer, D. W. (1988). Enhancing Student Learning: Emphasizing Essential Competencies in 
Academic Programs. Wilkes-Barre, PA: King’s College. 

Both Alvemo College in Wisconsin (www.alvemo.edu) and King’s College in 
Pennsylvania (www.kings.edu) were early postsecondary innovators incorporating core 
competencies into their curricula over 20 years ago. Alvemo ’s faculty restructured their 
institution around eight abilities each student should have: communication ability, 
analysis, problem solving, decision making, social interaction, global perspectives, 
effective citizenship, and aesthetic responsiveness. They couple their focus on abilities 
with student assessment that is authentic and reality-based. King’s College came to 
large-scale curriculum reform as a result of working on a planning project. In their 
CORE curriculum taken by all students, they seek to instill the skills of liberal arts 
learning: critical thinking, effective writing, effective oral communication, library and 
information literacy, computer competence, creative thinking and problem solving, 
quantitative reasoning, and moral reasoning. Farmer presents a thorough description of 
the hurdles, as well as the benefits, of the new approach. 

Jones, E. A. (Ed.). (1996). Preparing Competent College Graduates: Setting New and Higher 
Expectations for Student Learning. New Directions for Higher Education, No. 96. San 
Francisco: Jossey-Bass Publishers. 

Jones gives a summary of the recent press for “competent” college graduates in the first 
chapter of this monograph. Subsequent chapters are devoted to comprehensive reviews 
regarding what competencies are generally agreed upon in speech communication and 
listening, effective writing, critical reading, and problem solving. 



Career and Professional Education 

Gonczi, A. (1994). Competency-Based Assessment in the Professions in Australia. Assessment 
in Education, 1(1), 27-44. 

This article is a good introduction to the entire competency-assessment process. The 
section included on policy issues is comprehensive, focusing primarily on asking 
questions, not necessarily answering them. Two case studies included in the article give 
real-world examples of competencies, how they were assessed, and what the criteria were 
for evaluation of assessment results. These are good, practical, and short examples of 
implementation. 

Paulson, K., and Ewell, P. (In press). 21 sl -Century Skills for Community College Education: The 
Critical Role of Competencies. Mission Viejo, CA: League for Innovation in the 

Community College. 

Paulson and Ewell advocate using competencies to restructure community college 
education across the three traditional curricula of developmental education, general 
education, and vocational education. They discuss the pressures for competencies, 
benefits of using competencies for both students and institutions, how to determine 
competencies, and opportunities for community colleges to use competencies. The 
League’s Web Site is www.league.org 
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Competency-Based Curricula 



Sinclair Community College. (1998). Continuous Improvement Through the Assessment of 
Student Learning Outcomes: A Work in Progress — Department Assessment Reviews and 
Program Learning Outcomes. Dayton, OH: Sinclair Community College. 

This document provides a good example of an institution whose faculty have grappled 
with how to determine student learning outcomes. For each department, the document 
lists the guidelines and existing outcome lists referenced, student learning outcomes, a 
cross-listing of those outcomes with existing courses, assessment methods, the results of 
assessments, what institutional and departmental action has been taken based on those 
results, and perspectives on general education and overall outcomes. 

Western Governors University. A Proposed “Academic Infrastructure ” for Credentialing at the 
Western Governors University (WGU) [On-line]. Available: 
www.wgu.edu/wgu/about/acad_infra.html 

This working document was used in the early stages of developing Western Governors 
University (WGU). It gives the original conceptualization of the competency-based 
credentials offered by the institution, as well as discusses the implications for practice in 
terms of the structure of the institution, staffing requirements, and how students would 
progress through WGU programs to successful degree completion. 



Student Information on Expectations and Educational Progress 

Banta, T. W., and Associates. (1993). Making a Difference: Outcomes of a Decade of 
Assessment in Higher Education. San Francisco: Jossey-Bass Publishers. 

Banta, T. W., Lund, J. P., Black, K. E., and Oblander, F. W. (1996). Assessment in Practice: 
Putting Principles to Work on College Campuses. San Francisco: Jossey-Bass 
Publishers. 

These two books by Banta and colleagues are compilations of institutional assessment 
programs; some of these programs are based on student demonstration of competence. In 
Assessment in Practice, each major heading (such as “Assessing Student Achievement in 
the Major”) begins with a “chapter guide” that lists the assessment programs contained in 
the chapter “by institution, by subject area, by assessment method, and by case 
strengths.” These volumes are good for identifying institutions that are either currently 
using competencies or might have tried and decided against the use of competencies in 
their educational programs. Additionally, “assessment centers” are often used in the 
documentation of student competence; Making a Difference , Chapter 15, includes the 
pros and cons of such an approach. 

Meade, J. (1998). A Solution for Competency-Based Employee Development. HR Magazine , 
(December): 54-58. 

This product review for a human resources magazine highlights a software package that 
(1) tracks employees’ competencies, (2) monitors how those competencies match job 
requirements, (3) links to available online training materials, and (4) shows whether 
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employees are making progress in their training for the competencies. Postsecondary 
institutions might start documentation of student competency with something similar. 



Part IV. Competencies for Exit from Postsecondary Education 

Workforce skills are the primary focus of competencies used for exit from postsecondary 
education. This section includes citations on basic and workforce skills, performance 
certification, how to match individual skills with particular job requirements, and the skills 
necessary for admission to and success in professional fields. 



Basic Skills 

Comprehensive Adult Student Assessment System (CASAS). (1997). Extending the Ladder: 
From CASAS to Work Keys Assessments: Executive Summary [On-line], Available: 
www.nifl.gov/CASAS-ACT.html 

This executive summary offers the results of a study done linking two systems — CASAS’ 
Workforce Learning Systems, a basic adult literacy assessment system, and Work Keys, 
an assessment of employability skills. The article has a profile of the two systems that 
includes examples of assessment criteria used in both and practical descriptions of how 
the two systems are meshed together (the “extension ladder effect”). This overlap of 
systems allows them to collectively address a broader range of competence. This 
increased range provides a service to individuals who enter the basic literacy system and 
need to articulate to a more advanced skill level. Both systems use “real-life” workforce 
tasks, as opposed to “academic tasks.” The study used data from 494 individuals who 
were randomly selected from 27 sites in eight states. The authors conclude that 
Workforce Learning Systems is suitable for basic skill levels, the mid-level of skills are 
addressed by either system, and that higher skill levels are best assessed using Work 
Keys. (See also ACT’s Work Keys Targets for Instruction. (1994). American College 
Testing Program: Iowa City, IA. Books are available in the following subject areas: 
Writing, Applied Mathematics, Teamwork, Observation, Locating Information, 
Listening, Applied Technology, and Reading for Information.) 



Certification of Performance 

Browning, A. H., Bugbee A. C., Jr., and Mullins, M. A. (Eds.). (1996). Certification: A NOCA 
Handbook. Washington, DC: National Organization for Competency Assurance. 

This volume is an excellent introduction to certification. Of particular interest to those 
interested in competency-based initiatives are the chapters “Job Analysis” (Chapter 2), 
“Standard Setting” (Chapter 5), and “Future Trends in Credentialing” (Chapter 9). 
Chapter 2 provides a clear and unambiguous description of how to decompose a job into 
various tasks. The chapter on standard setting outlines the various theories of 
establishing standards for performance and methods for arriving at those standards. 
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Pottinger, P. S. (1979). Competence Assessment: Comments on Current Practices. In Defining 
and Measuring Competence, edited by P. S. Pottinger and J. Goldsmith. New Directions 
for Experiential Learning, No. 3, 25-39. San Francisco: Jossey-Bass Publishers. 

This chapter from the late 1970s focuses on licensing and registration based on 
demonstration of professional competence. It gives a short background introduction to 
professional competence and foreshadows many of the problems that continue to arise 
today, such as the reductionism inherent in long lists of behaviors and the need for 
multiple exemplars of competence. 



Matching Skills and Job Requirements 

Resnick, L. B., and Wirt, J. G. (Eds.). (1996). Linking School and Work : Roles for Standards 
and Assessment. San Francisco: Jossey-Bass Publishers. 

Resnick and Wirt’s volume is based primarily on SCANS and New Standards work. For 
the purposes of this bibliography, the most important section is Part One: Standards- 
Based Education for Workplace Readiness. The four chapters contained here are all 
useful, especially John H. Bishop’s chapter “Signaling the Competencies of High School 
Students to Employers,” because it is a practical approach to linking competencies with 
seeking employment. 



Professional Education 

Curry, L., Wergin, J. F., and Associates. (1993). Educating Professionals: Responding to New 
Expectations for Competence and Accountability. San Francisco: Jossey-Bass 
Publishers. 

Hagedom, L. S., and Nora, A. (1996). Rethinking Admissions Criteria in Graduate and 
Professional Programs. In Assessing Graduate and Professional Education: Current 

Realities, Future Prospects , edited by J. G. Haworth. New Directions in Institutional 
Research, No. 92, 31-44. San Francisco: Jossey-Bass Publishers. 

This chapter is a quick summary of graduate admissions procedures, including a 
discussion of alternatives. Hagedom and Nora highlight Stark et al.’s study of 
professional competence and ask the question, “How can [these competencies] be 
predicted?” Their conclusion is that more complex measures such as writing samples, 
plans for research, and structured interviews are necessary for determining graduate 
admission, but they do not go so far as to articulate admissions competencies. In a search 
to find institutions using alternative forms of admissions criteria, only a handful of 
institutions could be identified. Within those institutions, the departments using 
alternative criteria were primarily in the social sciences. These techniques included 
differential weighting of factors, portfolios, two-tier admissions processes (two onsite 
visits), and group interviews. 
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Stark, J. S., Lowther, M. A., and Hagerty, B. M. K. (1986). Responsive Professional Education: 
Balancing Outcomes and Opportunities, ASHE-ERIC Higher Education Report, No. 3. 
Washington, DC: Association for the Study of Higher Education. 

Stark, Lowther, and Hagerty ’s monograph from the 1980s documents the results of an 
extensive literature review. From their review, the authors identified six professional 
competencies — conceptual, technical, contextual, interpersonal, integrative, and 
adaptive — and five professional attitudes — career marketability, professional identity, 
ethical standards, scholarly concern for improvement, and motivation for continued 
learning. Curry, Wergin, and their associates build on the work of Stark and her 
colleagues in their 1993 volume elaborating on the concept of professional competency. 
The first section sets the stage for the renewed urgency for competencies. The second 
section highlights how competencies allow distinctions to be made in professional 
education for (1) linking to actual practice, (2) liberal learning, (3) ethics, and (4) critical 
thinking and problem solving. The third and final section discusses areas of praxis for 
the intelligent use of professional competencies in the 21 st century. 



Workforce Development/Vocational Education 

Evers, F. T., Rush, J. C., and Berdrow, I. (1998). The Bases of Competence: Skills for Lifelong 
Learning and Employability . San Francisco: Jossey-Bass Publishers. 

This book pulls together a number of issues not previously covered in a single volume. 
The first part, “Understanding Competence,” includes chapters on “the humbling effect” 
that occurs when recent college grads/new employees realize that they are unable to 
apply what they learned at college. The second chapter offers a common language for 
use with competencies. Part Two outlines the four “Essential Skills and Competencies” 
the authors have identified — managing self, communicating, managing people and tasks, 
and mobilizing innovation and change. Part Three is the useful essence of this book: 
“Developing Competence.” Three chapters explain (1) how the transition from college to 
workplace needs attention from both academics and businesspeople, (2) how colleges 
should be vigilant in their competency-based curricula and courses, and finally (3) how 
employers can be alert to the effects of their organizational interactions with recent 
graduates as well as their partnerships with postsecondary education institutions. Part 
Four consists of three case studies illustrating the principles outlined in the earlier 
sections. 

Packer, A. (1998). The End of Routine Work and the Need for a Career Transcript. Presented at 
the Hudson Institute’s Workforce 2020 Conference, October 7. [On-line]. Available: 
www.hudson.org/wf2020/wp98/pdf/SESSION3.pdf 

Packer discusses shifts in the match between skills that employees bring and the skills 
necessary for meaningful, high-paid work in the future economy. He then links this to 
the educational challenges created by these skill shifts. Finally, he suggests the structure 
of a “career transcript,” which is to “provide employers with readable, comprehensible, 
believable, timely and convenient documents [emphasis in original] that will help them 
choose the right applicant for the job.” The career transcript would include standardized 
tests, workplace performance assessments, and “structured assessment of classroom 
performance on benchmarked classroom tasks [emphasis in original],” although 
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presumably, based on his earlier arguments, “classroom” is not a necessary component of 
this latter requirement. 

Stecher, B. M., Rahn, M. L., Ruby, A., Alt, M. N., and Robyn, A. (1997). Using Alternative 
Assessments in Vocational Education. Santa Monica, CA: RAND. 

Stecher et al. write about vocational education that uses assessments to (1) improve 
learning and instruction, (2) certify individual mastery, and (3) evaluate program success. 
They recognize changes have occurred in vocational education resulting in a broader 
continuum of skills ranging from general workplace skills to very specific occupational 
skills. They point out that to evaluate skill acquisition, students must be assessed in a 
work context because skills are not enacted in isolation. This book contains a good 
introduction to the use of assessment for certification and in the validation of vocational 
education. (Note especially Appendix E, which is a synopsis of the Oklahoma 
Department of Vocational Technical Education Competency-Based Testing Program.) 

U.S. Department of Labor, Office of Policy and Research. (1998, July). Workforce Development 
Performance Measures Initiative: Final report [On-line]. Available: 
www.wdsc.org/transition/measure/fmalreport.htm 

This report offers the findings of a project. One goal of the project was to determine 
what measures could be used by a diverse set of constituents in employment, training, 
and education programs to track workforce development success. The second goal was 
to determine standard definitions for those measures. The main areas in which these 
measures would be used are (1) basic adult education, (2) occupational skills, and (3) 
employment/re-employment skills. The final list of performance measures includes nine 
core measures (such as basic skill attainment and transition success rate), nine “other 
measures of success” (such as starting wage at entered employment and diversity of 
occupations), and six developmental measures (including return on investment and 
system penetration rate). Appendix C includes the detailed descriptions of the 24 
performance measures. 



Part V. Competencies Used for Overall Institutional Effectiveness 

This section has as its focus the use of competencies by institutions for measuring how well an 
institution is meeting its goals. Included here are entries having to do with accountability, 
accreditation, and performance funding. 



Accountability Processes 

(Note: “Accountability Processes” is defined here as statewide student testing programs 
and measures of quality.) 

National Center for Higher Education Management Systems. (1996). The National Assessment 
of College Student Learning: An Inventory of State-Level Assessment Activities. A Report 
of the Proceedings of the Third Study Design Workshop. Boulder, CO: National Center 
for Higher Education Management Systems. 
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National Center for Postsecondary Improvement. (1997, May). Benchmarking Assessment: 
Assessment of Teaching and Learning in Higher Education for Improvement and Public 
Accountability: State Governing, Coordinating Board & Regional Accreditation 
Association Policies and Practices. Palo Alto, CA: Stanford University, College of 
Education. 

These two documents summarize statewide testing programs by state. NCHEMS’ 
compendium includes the proceedings from a workshop supplemented with 
documentation on “the extent and character of current state-level activities in assessing 
postsecondary student outcomes.” These characterizations include descriptions of the 
origins and development of the state’s program, assessment instruments used, obstacles 
to expansion of assessment activities, and what the state saw as the most “important 
needs to mov[ing] a national assessment agenda forward.” NCPI also summarizes state 
assessment activities to include contact information, analysis of state assessment policy 
including the policy context, policy type and what stage the policy was in at the time of 
publication, guidelines, indicators, instruments used, and assorted additional information. 
NCPI also includes the work of regional accrediting associations in their review. Singly 
each is a wealth of information; together the two documents form a powerful reference on 
statewide assessment and testing programs. 

Nordvall, R. C., and Braxton, J. M. (1996). An Alternative Definition of Quality of 
Undergraduate College Education. Journal of Higher Education, 67(5), 483-497. 

Nordvall and Braxton offer a response to traditional quality mechanisms built on 
reputational, resource, and value-added measures. They seek to drive the quality process 
back into the classroom and to the level of student engagement using Bloom’s taxonomy. 
Their approach rests heavily with faculty and eliminates student motivation issues that 
are a problem with assessments “added-on” to the curriculum. Although they never use 
“competency” by name in their ideas, they allude to it when they describe what they 
mean as “the level of understanding of course content to be demonstrated by students 
while engaging in course-level processes” (p. 486). A short discussion on the problems 
of aggregating course and classroom-level information to the department and institution 
level is included. 



Accreditation 

(Note: The Council for Higher Education Accreditation (CHEA) and the Western 

Association of Schools and Colleges (WASC) are working on standards that incorporate 
student competencies, but these efforts are not public yet.) 

Ewell, P. T. (1998). Examining a Brave New World: How Accreditation Might be Different. 
Speech presented at the 2 nd CHEA “Usefulness” Conference. [On-line]. Available: 
www.chea.org/Events/Usefulness/98May/98_05Ewell.html 

Ewell focuses on the state of accreditation and how it might continue to evolve. Three 
main themes frame his talk: (1) the revolution in teaching and learning, (2) the “de- 
institutionalization” of learning, and (3) the need for public engagement. He calls for 
“examining the integrity of the degree” using learning outcomes as the mechanism for 
this investigation. He notes that it is not merely whether the degree is of quality but how 
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resources and processes are “configured and used” to create a quality degree that must be 
considered. Practical suggestions for how to implement these ideas are given. 

Teacher Education Accreditation Council, (n.d.). Accreditation Goal and Principles of the 
Teacher Education Accreditation Council (TEAC). [On-line], Available: 
http://www.teac.org/accreditation/goals/index.asp 

TEAC’s three Quality Principles are (1) evidence of student learning, (2) valid 
assessment of student learning, and (3) institutional learning. The cycle created by these 
three principles, if done well, incorporates the need to explicitly state what student 
learning is expected, how that learning will be assessed, whether the assessment is valid 
in the eyes of the program faculty (and presumably to the groups that employ their 
graduates), and how that information is used to improve the program in terms of 
“institutional learning.” 



Performance Funding 

Banta, T. W., Rudolph, L. B., Van Dyke, J., and Fisher, H. S. (1996). Performance Funding 
Comes of Age in Tennessee, Journal of Higher Education, 67(1), 23-45. 

This short historical profile of Tennessee’s performance funding process is linked to 
survey data collected from all 23 performance-funding coordinators in Tennessee. The 
survey focused on the usefulness of the various performance funding measures. 
“Competencies” are not specified by the state, but they are implied by the use of general 
education assessment and major field testing requirements in the performance funding 
criteria. Survey respondents noted that while these testing efforts have some effect on 
changing programs, it was also apparent that there is little motivation for students or 
institutions to take them seriously because no direct path exists linking competencies and 
assessments to student feedback and program improvement. (See also Banta, T. W. 
(Ed.). (1986). Performance Funding in Higher Education: A Critical Analysis of 

Tennessee’s Experience. Boulder, CO: National Center for Higher Education 

Management Systems.) 

Burke, J. C., and Serban, A. M. (Eds.). (1998). Performance Funding for Public Higher 
Education: Fad or Trend? New Directions for Institutional Research, No. 97. San 
Francisco: Jossey-Bass Publishers. 

This volume is the most comprehensive and recent monograph on performance funding. 
With regard to competency-based initiatives, however, it is rather sparse. A close reading 
of individual performance indicators reveals some measures that might nominally be 
linked to student competency. Some examples of potential indicators include “indicators 
of the effectiveness of remediation,” “pass rates on professional exams,” “job placement 
of graduates,” and “employer surveys.” They conclude that aggregating individual 
student competency and achievement to arrive at program or institutional measures of 
effectiveness has not yet been done effectively or equitably in performance funding. 
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7. CASE STUDY ABSTRACTS 



KING’S COLLEGE CASE STUDY 
ABSTRACT 

In the fall 1999 semester, 18 faculty from different disciplines were interviewed at King’s 
College. The purpose of this case study was to explore the creation of competencies and how 
they are assessed at King’s College. 

King’s College is an independent baccalaureate institution that offers undergraduate educational 
programs in the humanities, the natural and social sciences, and specialized programs in business 
and other professions. King’s College is rooted in the tradition of Judeo-Christian humanism and 
seeks to educate the entire person. 

The general education curriculum at King’s College during the 1970s was primarily a set of 
distribution requirements giving students numerous choices. The curriculum was perceived by 
many faculty to be a smorgasbord of learning with fragmented experiences that lacked strong 
connections among individual courses. As faculty grew increasingly dissatisfied with the 
distribution form of general education, they decided to design a new general education core 
curriculum with a particular focus on student outcomes. Faculty project teams designed and 
implemented a core curriculum to provide students with a common learning experience in the 
liberal arts and sciences. General education courses were created to address eight different skills 
of liberal learning. Each liberal learning skill, such as effective writing, began with a stated goal 
and then specified objectives associated with the achievement of the broader goal. 

Faculty developed competency growth plans that included a definition of each liberal learning 
goal or competency, an indication of courses and assignments that were designed to help students 
develop each competency (across the entire baccalaureate degree program), and specific criteria 
that were used to gauge the quality of student performance. Faculty in all academic programs 
articulated these dimensions and reinforced key ideas in course syllabi and instructions for 
assignments. 

Assessment at King’s College was primarily course-embedded and provided students with clearly 
defined expectations, personalized feedback on growth, and timely indications of areas needing 
attention. Faculty obtained the information needed to identify and respond to the strengths and 
weaknesses of individual students, of teaching/learning strategies, and of curricula. The 
assessment program helped faculty create an integrated plan for cumulative learning. The faculty 
rejected standardized, longitudinal testing of student learning using commercially available 
instruments when they found that students did not take these types of external assessments 
seriously, and the tests did not measure all the areas they considered to be important. Course- 
embedded assessments motivated students to take these evaluations seriously and to perform at 
their highest levels because the outcomes counted toward their grades. Faculty designed their 
own assessments, including the sophomore-junior diagnostic project and the senior integrated 
assessment. 
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NORTHWEST MISSOURI STATE UNIVERSITY CASE STUDY 

ABSTRACT 



In the spring 2000 semester, seven faculty were interviewed at Northwest Missouri State 
University (Northwest). Participants were predominately full-time faculty who were directly 
engaged in or provided leadership for assessment. The purpose of this case study was to explore 
the competency-based educational system at Northwest with a particular focus on the 
development of competencies and how they are assessed. 

Northwest is a moderately selective, learner-centered regional institution offering a range of 
undergraduate and graduate degree programs. In 1992, Northwest faculty and administrators 
adopted the Malcolm Baldridge National Quality Award criteria as a conceptual framework to 
discuss quality and as a template for planning and agenda setting. This action was introduced as a 
master plan to revitalize undergraduate education. The culture of quality plan was developed by 
identifying the 42 best practices and resulted in specific strategies to improve processes. Success 
at Northwest is measured by the value that is added to students, faculty, staff, and the region. 

The Coordinating Board for Higher Education (CBHE) in Missouri oversees higher education for 
the entire state. A major goal of CBHE is to provide an educational system that will respond to 
students’ needs and facilitate easy mobility across institutions. In 2000, CBHE revised statewide 
general education policies that directly influence general education at Northwest. These policies 
include an articulation of competencies that serve as specific examples of state-level expectations 
for student performance in general education. The competencies include specific sets of skills in 
communicating, higher order thinking, managing information, and valuing. The knowledge 
competencies focus on social and behavioral sciences, humanities and fine arts, mathematics, and 
natural sciences. Faculty at Northwest designed their general education program to fit their 
mission and will be reexamining how it meets the state-level curricular goals. 

Northwest faculty follow a seven-step planning process that applies to general education as well 
as to all major programs. Through this process, they define major objectives or key quality 
indicators that are based on customers’ needs and use multiple measures to determine whether 
these outcomes are being achieved. For example, faculty use the Academic Profile to assess 
general education outcomes. All undergraduates take a locally developed assessment at the end 
of their second writing course. Students are given a series of short, related readings about a 
controversial topic and then asked to respond to specific prompts. The Missouri Colloquium on 
Writing Assessment is a statewide organization that exists to support faculty members in all 
colleges and universities within the state. This group surveys institutions each year to identify the 
types of assessment methods used. 

In the future, Northwest will be developing an electronic portfolio that can both include 
assessments of students’ out-of-class experiences and allow for all assessment results to be posted 
electronically. 
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SINCLAIR COMMUNITY COLLEGE CASE STUDY 
ABSTRACT 



In the fall 1999 semester, four faculty members were interviewed at Sinclair Community College. 
The purpose of this case study was to explore the development of competencies and how they are 
assessed. Sinclair Community College is a comprehensive 2-year institution offering a diverse 
range of parallel university, technical, and career courses to students. It is the largest single- 
campus community college in Ohio and among the 20 largest in the United States. In the mid- 
1980s, Sinclair Community College administrators and faculty began considering the possibility 
of guaranteeing the entry-level skills of their graduates. They realized that before they could 
make such a guarantee, they should first clearly articulate student learning outcomes and 
determine if they were meeting employers’ needs. This was the first step toward defining 
competencies and developing an assessment plan at Sinclair. In 1992, Sinclair decided to 
guarantee the entry-level skills for their graduates. 

Sinclair, embracing the principles of the learning college, wanted to move toward more active and 
collaborative experiences for their students. Faculty believed substantial changes that would lead 
to better retention of material across courses and to help students reach higher levels of learning 
were needed. 

The faculty have identified specific competencies for general education and designated courses to 
fulfill the seven different competencies. In effect, these competencies serve to define the specific 
knowledge and technical skills that college students should be able to demonstrate. 

As a result of the open-door admissions policy at Sinclair, the abilities of entering groups of 
students vary substantially. Therefore, the college measures the entry-level competencies of 
students through Computer Adaptive Placement Testing (COMPASS) developed by the 
American College Testing Company. Faculty also developed checklists for writing and 
communication that served as criteria for evaluating student assignments within individual 
courses. For example, the writing checklist included an evaluation of assignment profile, content, 
organization, style, and mechanics. Each academic department at Sinclair has developed or 
refined multiple methods to assess student achievement of specific learning outcomes within their 
majors. These assessment plans are open to revisions over time. There are a variety of 
assessment methods that academic program faculty use, including commercially available and 
standardized national exams, comprehensive exams and final projects evaluated by peers, faculty 
and/or external evaluators, comprehensive research papers or theses, programmatic growth 
contracts, observations, and simulated work experiences. 

A recent initiative underway at Sinclair is labeled “process learning.” Some faculty are defining 
specific competencies for individual courses in applied critical thinking, problem-based learning, 
cooperative teams, and communications. 
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HAGERSTOWN COMMUNITY COLLEGE CASE STUDY 

ABSTRACT 



In the fall 1999 semester, six faculty were interviewed at Hagerstown Community College 
(HCC). Participants were mostly full-time faculty who were directly involved and provided 
leadership in the development and implementation of the competency-based educational 
experiences. The purpose of this case study was to explore the career transcript system used at 
HCC that builds upon earlier work completed on CD-ROM modules. 

HCC is part of a consortium consisting of several community colleges, related industries, and a 
multimedia production company. This consortium was originally created in 1995 under the 
leadership of the Johns Hopkins University to better prepare community college graduates for the 
21 st century and to help students learn important competencies so that they would remain valuable 
employees. The consortium also wanted to provide students with a degree that would be 
recognized by employers in different states. 

They initially worked on developing CD-ROMs to integrate workplace competencies into general 
education coursework. Ultimately with funding from NSF, the consortium focused on covering 
the Secretary’s Commission on Achieving Necessary Skills (SCANS) through five CD-ROM 
modules assigned to a particular academic home. Each module was embedded into the 
appropriate existing course. One module was created for math, three modules for science, and 
one module for a communication course. The CD-ROMs and accompanying materials for each 
module constituted about 9 hours of a particular course or 20 percent of a three-credit semester 
course. For each particular module, faculty identified one or two primary SCANS skills that were 
addressed. 

Faculty worked with AES, International (a commercial company) to develop behavioral 
indicators to demonstrate performance mastery of the SCANS skills. Instructors prepared lists of 
tasks and the actual behaviors they expect of students completing the projects as outlined in the 
CD-ROM learning experiences. Then AES translated these behaviors into an assessment scoring 
rubric. Faculty were taught how to use the constructed rubrics as a means of documenting 
student performance. 

As this new system incorporating project-based modules was embedded into the curriculum, 
faculty and employers strongly believed that the traditional academic transcript was not sufficient 
to demonstrate students’ mastery of key concepts and SCANS competencies. Therefore, a career 
transcript was designed by a group of faculty and a commercial company to indicate students’ 
specific levels of achievement with support from NSF. In 1999, project leaders began testing this 
Internet-based career transcript. The main assessment results from the modules were placed into 
the Internet database. Students could review and print out their transcripts and decide what 
information they wanted to share with employers. This career transcript indicated the specific 
competencies that students had mastered and supplemented the traditional transcript. Organized 
like a resume, it listed the SCANS skills taught and levels of achievement. 





COMMUNITY COLLEGES OF COLORADO 
INCUMBENT WORKER PROJECT 
CASE STUDY ABSTRACT 



The purpose of this case study was to explore the beginning development of the Incumbent 
Worker project funded by the U.S. Department of Labor as 1 of 12 national demonstration grants. 
A consortium of community colleges and businesses are formally collaborating to address the 
employment needs within the state of Colorado. This site was selected because the training 
initiatives will include competencies and performance objectives for employees. 

The Community Colleges of Colorado (CCC) is providing leadership for this 2-year endeavor 
that involves 14 partner colleges with the local governing entities under a central office 
comprising an Executive Director and staff. Each of these colleges has a training division and an 
instructional division that designs and delivers learning to corporate and small business 
customers. 

This grant has two major purposes. First, training services will be offered directly to incumbent 
workers at certain companies. Second, a state-level college training support system will enhance 
local company/employee services. Leaders of this grant expect that at least 4,350 incumbent 
workers will receive necessary services. Each community college is in a strong position to work 
collaboratively with industries, since they each have a business service unit on campus that 
includes experienced assessment and training staff as well as managers. 

A major portion of the grant is dedicated to creating the Colorado State Training Solutions 
Center. This Center is envisioned to address employment issues by overseeing the creation, 
revision, and delivery of training modules needed by different companies. The Center will utilize 
an electronic storage and retrieval system so that their partners can have access to a wide range of 
materials for their use and modification. Since these materials are modularized and their delivery 
formats will be standardized, partners will be able to draw upon and assemble different types of 
modules into new course offerings. This system will foster a competency-based, modularized 
curriculum that supports both group and independent-learner models with various formats 
including instructor-led classroom training, Web-based instruction and performance support, CD- 
ROM, video, and audio. 

The leaders of this initiative believe that the curriculum will serve as a model since it defines the 
creation and delivery of learning content through stated expectations of learning and performance 
and provides for the measurement of the results. Assessments are designed by the training 
curriculum designers and are intended to measure student learning before and after the delivery of 
modules. The types of learning assessments will include paper-and-pencil tests or demonstrations 
of learning through applied tasks. During the training experience, participants will be asked for 
feedback about whether the training material matches their needs. If it does not match, then the 
trainer will be able to revise the learning experiences to better fit employees’ needs. 
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WESTERN GOVERNORS UNIVERSITY CASE STUDY 
ABSTRACT 



In the fall 1999 semester, five individuals were interviewed at Western Governors University 
(WGU). The purpose of this case study was to explore the competency-based degree programs in 
place at WGU. WGU is a newly incorporated institution that has dual missions for delivering 
competency-based degrees and brokering of distance-delivered education. 

In 1995, governors of most of the western states working in conjunction with the Western 
Governors Association agreed to develop an accredited postsecondary educational entity. It 
would be designed to broker an array of distance-delivered learning opportunities from both 
traditional postsecondary providers and corporate providers to students across the country and the 
world. WGU offers competency-based programs of study that are designed to recognize 
knowledge and skills learned through work and life experiences. The certificates and degrees, 
especially those developed initially, support welfare-to-work initiatives as well as create 
opportunities for the geographically dispersed clientele of WGU. Their competency-based 
degrees actively allow students to integrate past experience and use it toward degree completion. 
It is believed that this accelerates student progress to graduation. 

All WGU degrees are competency based and are predominately at the Associate of Applied 
Science and Associate of Arts levels. These degrees are most needed in the western region of the 
United States. The basic building blocks of WGU degrees are the individual competencies that 
consist of performance descriptions. These are the smallest unit of describable skills or 
knowledge upon which everything else is based. A domain consists of a collection of 
performance descriptions that corresponds to a body of knowledge roughly equivalent to a 
sequence of courses in a traditional degree program. Similarly, several domains (usually six or 
seven) compose a WGU degree. The initial developers of the WGU degree structures found that 
using competencies allowed for greater flexibility in program and degree design, making possible 
the linkage of subject matter in fresh combinations. 

Program Councils comprising academic and corporate subject matter experts who oversee 
clusters of credentials in specific areas are responsible for the development of the curriculum. 
The Assessment Council evaluates student learning and progress, while the advisors are 
responsible for the development of the curriculum and mentors provide advising. 

WGU seeks to integrate competencies during assessment, and therefore an assessment battery is 
typically used. This collection of assessment instruments is often developed by third parties and 
includes customized assessments built to WGU specifications. WGU follows accepted 
professional practice for establishing reliability and validity of these new instruments. A method 
of tracking student performance in the future and asking employers about their abilities is being 
planned. 
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PROFICIENCY-BASED ADMISSION STANDARDS SYSTEM (OREGON) 

CASE STUDY ABSTRACT 



This case study focuses on the Proficiency-based Admission Standards System (PASS) in the 
state of Oregon. The impetus for adopting a competency-based admission system in Oregon is to 
build on and integrate with K-12 systems that are increasingly competency driven. 

In 1991 and 1995, the Oregon legislature adopted the use of the Certificate of Initial Mastery 
(CIM) at the 10 th grade level and the Certificate of Advanced Mastery (CAM) at the 12 th grade 
level. Both of these certificates focus on competencies necessary for student achievement. CIM 
requires achievement in English, math, social sciences, arts, and a second language. CAM 
requires achievement in the same academic areas plus generic, career-oriented competencies in 
personal management, problem-solving teamwork, communication, workplace systems, career 
development, and employment foundations, as well as application of academic knowledge in a 
work context. In response to these changes at the secondary level, the Oregon University System 
(OUS) in 1994 adopted a proficiency-based method for admission to colleges and universities in 
its system. PASS is to be phased in over a number of years, with full implementation expected 
with the entering class of 2005. 

Oregon University System chose to adopt a proficiency-based system to increase the likelihood 
that students who are admitted to college will be able to succeed and to graduate in a timely 
fashion. Another reason to move toward this competency-based system is the increasing 
dissatisfaction with the reliability of grade point averages as proxies for student learning. 
Advocates of this new system believe that over time, proficiencies will be better measures of 
student learning. In addition, administrators would like to use PASS proficiencies and scoring in 
placement decision, too. Another suggested future outcome is that the Associate of Arts Oregon 
Transfer degree will eventually be defined by proficiencies rather than the credits now used, 
removing obstacles to seamless transitions along the path to the baccalaureate degree. 

When fully implemented, PASS will require students to demonstrate proficiency in six subject 
areas — math, English, social science, science, visual and performing arts, and second languages. 
For each of these subject areas, there are 4 to 7 individual proficiencies for a total of 33 outcomes. 
PASS proficiencies are what students should know and be able to do. 

Concerns of parents, students, and admissions officers center on transfer and articulating across 
postsecondary systems. These have been addressed by working with major out-of-state 
institutional receivers and providers of students. Agreements in principle have been reached with 
the University of California and California State University systems, the University of 
Washington, and private universities such as Stanford that have provided letters of support to 
PASS. Additional concerns focus on the award of financial aid and scholarship monies that are 
based on grade point averages and credits, as well as other requirements. Steps are being taken to 
address these issues. 
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FORD MOTOR COMPANY CASE STUDY 
ABSTRACT 



Five individuals from the Human Resources unit in the Ford Motor Company Headquarters in 
Dearborn, Michigan, were interviewed about their competency-based selection and training 
system. The Ford Motor Company produces cars and trucks. The company and its subsidiaries 
also engage in other businesses, including manufacturing automotive components and systems as 
well as financing and renting vehicles and equipment. 

Ford Motor Company uses competencies in their selection, training, and promotion of global 
salaried employees. Ford defines competencies as an agreed-upon set of knowledge, skills, 
abilities, experiences, values, or personal characteristics that are necessary to run the business and 
achieve business objectives. The competency model that Ford developed was based upon a 
formal review of existing models in other companies. They performed an analysis of effective 
behaviors, particularly for managers. These competencies are clustered into three main areas that 
are considered important for leadership behaviors: (1) heart, (2) relationship, and (3) know-how. 
Each cluster is briefly defined by a few key terms. Then, there are four subcategories under each 
major category that further define specific competencies. Within the Human Resources System 
Infrastructure, these competencies guide several processes at several points in time. First, 
competencies are used as a basis to screen individuals who are recruited for positions. Second, 
competencies are formally assessed to identify who will ultimately be selected or hired to fill 
specific positions. Competencies are used to evaluate the performance of management. They 
also guide the design of training seminars and workshops. Finally, competencies are used to 
create development plans for leaders who aspire to make a lateral move within the company or 
seek a promotion. 

Assessment of individuals’ abilities at Ford begins with the recruitment of potential employees 
considered for professional positions. . All candidates who pass the initial pre-screening must 
complete a job simulation that lasts about 3 hours. All of these assessments are designed by Ford 
employees and tailored to the specific job function as well as the competencies. Each team of 
candidates progresses through multiple assessments including a case of an actual business 
problem, a stressful role-play situation, and a group discussion about a particular problem. 
Professional staff members in an assessment center are trained for 2 days to learn how to evaluate 
the candidate’s performance. At least two assessors evaluate each simulation, and a more highly 
trained coordinator supervises each pair of assessors. The coordinator is responsible for 
reviewing their assessments and then working with the raters to resolve any major differences in 
their ratings. There are no cutoff scores that are needed by candidates in order to be hired at 
Ford. However, if candidates scored at the very ineffective levels across all simulations and 
assessment, they most likely would not be hired. The Human Resources staff believe their 
training of assessors strengthens the reliability of their results. In addition, they are making 
progress toward predictive validity since they use actual problems that are encountered at Ford as 
the major assessments. The assessment system has not been in place long enough to reach strong 
predictive validity, but Human Resources staff view this outcome as possible with more 
experience over time. 
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A. King’s College 

B. Northwest Missouri State University 

C. Sinclair Community College 

D. Hagerstown Community College 

E. Community Colleges of Colorado Incumbent Worker Project 

F. Western Governors University 

G. Proficiency-Based Admission Standards Systems (Oregon) 

H. Ford Motor Company 

I. Case Study Interview Protocol 
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APPENDIX A 



King’s College 



Introduction 

In the fall 1999 semester, 18 faculty were interviewed at King’s College by the Working Group 
consultant. Each individual interview during the 3-day site visit lasted between 50 to 60 minutes. 
Participants were mostly full-time faculty representing different ranks (full, associate, and 
assistant professors). In addition, faculty leaders such as department chairs, team leaders of 
curriculum design, and senior administrators were interviewed. Numerous documents were 
reviewed by the consultant, including the college catalogue, published books, and selected 
chapters about King’s College, as well as their most recent outcomes assessment periodic report 
submitted to the Middle States Association of Colleges and Schools. The purpose of this case 
study was to explore the competency-based educational system at King’s College with a 
particular focus on the development of competencies and how they are assessed. 



The Setting 

King’s College is an independent baccalaureate institution with an enrollment of 1,700 full-time 
students plus 600 part-time and graduate students. The student body consists of mainly 
undergraduates from Pennsylvania, New Jersey, New York, Delaware, Maryland, and 
Washington, D.C. Students also come from 14 different countries. 

King’s College offers primarily undergraduate educational programs in the humanities, the 
natural and social sciences, and specialized programs in business and other professions. King’s 
College is rooted in the tradition of Judeo-Christian humanism and seeks to educate the whole 
person (King’s College, 1999a). 

The general education curriculum at King’s College during the 1970s gave students many choices 
and only vaguely defined its educational philosophy (Farmer, 1988). The curriculum did not 
articulate specific learning outcomes expected of students. This curriculum was perceived by 
many faculty to be a smorgasbord of learning with fragmented experiences that lacked strong 
connections among the individual courses. 

Since faculty grew increasingly dissatisfied with the distribution form of general education, they 
decided to design a new general education core curriculum with a particular focus on student 
outcomes. The curriculum committee decided that this new curriculum should consist of all 
newly designed courses (Farmer, 1988). All new courses were created by faculty project teams 
comprising faculty from more than one discipline (Farmer, 1999a). The project teams were 
empowered to make recommendations and implement the changes into action. The academic 
vice president deliberately developed teams rather than committees because committees are often 
viewed by faculty as having an emphasis on protecting territorial interests of their own 
department or academic program (Farmer, 1999b). Frequently, committees make 
recommendations and then others actually implement the ideas. Teams are more fully committed 
to conceptualizing new ideas and then actually striving to incorporate actions into the culture of 
the institution. 
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Identifying Important Outcomes 



Faculty initiated their work by discussing and then reaching a consensus about the desired student 
learning outcomes. After these outcomes were defined, the faculty identified the course content 
most appropriate to link with those outcomes. 

In 1985, the King’s College faculty implemented a core curriculum designed to provide students 
with a common learning experience in the liberal arts and sciences, so that students could develop 
the following specific skills of liberal learning: 

• Critical thinking; 

• Effective writing; 

• Effective oral communication; 

• Library and information literacy; 

• Creative thinking and problem solving; 

• Moral reasoning; 

• Computer literacy; and 

• Quantitative reasoning. 



All courses in the core curriculum were designed for nonmajors (Farmer, 1988). The faculty 
believed that this new core curriculum would be vital to promote a stronger coherence and 
integrity of knowledge within this new structure. Each liberal learning category had clear, 
specific goals and objectives for all courses within it. These goals and objectives included 
numerous connections between the liberal learning categories (King’s College, 1999a). 

King’s College faculty believed that an undergraduate education helps students to view learning 
as cumulative, transferable, and integrated. Therefore, faculty developed competency growth 
plans for each academic program. They defined each transferable liberal learning skill within the 
context of the major and then divided the skill into specific competencies for students to develop 
from the freshman year through the senior year in both the general education core and major 
courses. Each plan included a definition of each competency, an indication of courses and 
assignments that were designed to help students develop the competency, and specific criteria 
faculty and students use to gauge the quality of student performance. Attachment A-l is an actual 
example from the accounting department that is currently used for their accounting majors. 
These plans are guides for faculty and students. The main concepts were frequently reinforced in 
syllabi and instructions for course assignments. 

Faculty derived these competencies mainly from their own experiences within the field; they 
learned how to write specific, clear criteria from special educational workshops that they have 
attended at King’s College. Faculty development was a serious endeavor, and professors were 
encouraged to participate in an ongoing series of seminars during the summer or a particular 
semester. The goal was to help faculty gain self-confidence and to be prepared professionally to 
implement their new ideas. Group training for faculty in computer literacy, critical thinking, or 
writing across the curriculum were some of the development work necessary for faculty to 
become responsible for the total education of their students (Farmer, 1988). 
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Assessment Strategies 

The main purpose of assessment at King’s College was to enhance student learning by focusing 
on improvements rather than comparing programs. The main questions that faculty sought to 
answer were: 

• What should students know? 

• How well are they learning it? 

• How does the institution know? 

King’s College adhered to the four tenets of the 1990 Middle States Association Framework for 
Outcomes Assessment: 

• Students hold the major responsibility for their own learning; 

• Teaching plays a significant role in facilitating student learning; 

• Campus climate plays an important role in promoting both learning and teaching, 
especially in terms of student development; and 

• Equity and diversity in campus life contribute to the success of teaching and learning 
(King’s College, 1999a). 

Assessment at King’s College was primarily course-embedded and provided students with clearly 
defined expectations, personalized feedback on growth, and timely indications of areas needing 
attention (King’s College, 1999a). Faculty obtained the information needed to identify and 
respond to the strengths and weaknesses of individual students, of teaching/leaming strategies, 
and of curricula. 

The assessment program helped faculty to create an integrated plan for cumulative learning. 
Faculty critically evaluated these cumulative plans systematically to determine if there was a need 
to update or revise them. For example, the faculty in the College of Arts and Sciences identified 
the problem of outdated computer competence growth plans (King’s College, 1999b). Each 
department was working on developing new technology growth plans to reflect the essential 
learning outcomes that all college students should achieve. These discussions also led to 
revisions in the curriculum, including the decision of the English department faculty to add an 
elective course in desktop publishing to their curriculum. 

The faculty traditionally rejected standardized, longitudinal assessments of student learning using 
commercially available instruments since they found that students do not take these types of 
external assessments seriously and they do not measure all of the areas faculty believe are 
important. Course-embedded assessments motivated students to take these evaluations seriously, 
and they are motivated to perform at their highest levels since the outcomes will count toward 
their grades (Farmer, 1999b). However, in the 1999-2000 academic year, commercially available 
standardized instruments will be pilot tested to determine if they address concerns raised by some 
faculty who want quantitative data to provide benchmarks or external reference points for making 
comparisons. The initial pilots will be focused on critical thinking, since it is a main priority of 
both general education and major programs (King’s College, 1999b). 
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King’s College faculty designed all of their own assessments. They administered assessments 
and evaluated student learning within individual courses. Faculty primarily used qualitative 
assessment strategies defined by criteria conceptualized as a developmental plan of learning for 
students. These criteria helped faculty judge student performance. Since qualitative assessment 
activities were actually designed by the faculty, they reflected more closely what was actually 
being taught by faculty and being learned by students (King’s College, 1999b). 

Several major components made up King’s College comprehensive assessment program. Faculty 
in the core curriculum designed course-specific assessments (either individually or collegially) by 
creating exercises to understand how well students think and communicate. These may be 
administered at the beginning and at the end of the general education core courses. Specific 
assignments within the context of courses may also be used for assessment purposes. For 
example, faculty may give students a pretest at the beginning of a critical thinking course and 
then incorporate an assessment activity as part of the final exam in the same course. 

The sophomore-junior diagnostic project was used by departments or programs as a screening 
exercise to determine each student’s ability in his/her major field of study regarding mastery of 
content and methods within the discipline. It also provided useful assessment of the student’s 
ability to apply the transferable liberal learning to a project within his or her major field of study 
(King’s College, 1999b). Faculty in each major designed a project that was related to the major 
field and is usually embedded in a required second semester sophomore or first semester junior- 
level course. This midpoint assessment enabled faculty to measure the student’s likelihood of 
success in the major. 

The culminating component of the King’s College curriculum was the senior integrated 
assessment (King’s College, 1999b). Faculty in each department created an activity that allowed 
students to demonstrate their mastery of subject matter and the methods of the major field as 
reflected in departmental goals of liberal learning articulated for seniors in the department’s 
competence growth plan. These assessment activities frequently included exhibits, lectures, or 
other kinds of public presentations where other students, faculty, and the public are invited to 
attend. All undergraduates completed this assessment, which served to demonstrate the students’ 
readiness to function at an effective level to meet the expectations of the King’s College faculty 
and the expectations of employers in the workforce. 



Documenting Student Achievement 

Faculty project teams for general education and departmental faculty for major programs were 
responsible for submitting assessment documentation. The reporting requirements included: 

• “A description of the assessment activity; 

• Criteria students are given in writing before the assessment; 

• Samples of student performance at three levels — superior, satisfactory, 
less than satisfactory but still acceptable — with names of students deleted, 
but with feedback to students included; 

• A report of the number of students performing at each level; and 

• An explanation of how assessment results will be shared with faculty on a project 
team or in the department” (King’s College, 1999b, p. 32). 
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The work of the core curriculum project teams was supervised by the Dean of the College of Arts 
and Sciences and was reviewed by the Curriculum and Teaching Committee of the Faculty 
Council. Some faculty developed common criteria for all sections of certain individual core 
general education courses. Effective assessment criteria should reflect the following 
characteristics: 

• “Communicate to students in writing avoiding ambiguous language; 

• Relate directly to faculty expectations for student learning at the appropriate level 
from freshman to senior year of study; 

• Reflect the significant aspects of learning in a discipline avoiding what is trivial or 
petty; 

• Are sufficiently specific in nature to permit student and teacher to monitor progress 
in building strengths and repairing weaknesses; 

• Provide the basis for giving feedback to students and encouraging a continuing 
discussion; and 

• Encourage students to engage in self-assessment and to take more responsibility for 
their own learning” (Farmer, 1999b). 



The project teams recorded across all sections of a course the number of students scoring on the 
assessment at the highest, mid-level, and “needs improvement” (King’s College, 1999b). Three 
copies of each assessment (representing each of the three levels) administered by a project team 
were retained with the specific criteria that faculty use to evaluate student performance. Project 
team members reviewed each other’s work for the purpose of enhancing inter-rater reliability 
based upon the use of common criteria. The members of the project team attempted to identify 
common student problems and used assessment results as a foundation to create strategies to 
strengthen student learning (Farmer, 1999b). Faculty in academic programs also followed this 
approach. They were supervised by their department chairpersons, and the relevant dean 
reviewed the assessment findings with chairpersons on an annual basis. 

The documentation material was available to external evaluators who could review and make 
judgments in arriving at consensual validation regarding whether the faculty had established 
criteria at a sufficiently high enough level to be appropriate at the freshman through the senior 
years. Secondly, it could be determined if faculty are honest in the application of their own 
criteria (King’s College, 1999b). The regular, cyclical collection of student assessment results 
gave project teams and departmental faculty opportunities to discuss the findings in relation to 
course content and sequence. 



Data Ramifications 

The majority of assessments of student learning were embedded within existing coursework both 
in general education and the major programs. Some faculty used common assessments in the 
general education courses, while others used different approaches. 

Most general education courses included a pre- and post-assessment that helped faculty to 
determine if there were gains in student learning. The assessment documentation required for all 
programs was available to external evaluators to review and to make judgments about the quality 
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of information. Such a process can help strengthen validity since external evaluators can examine 
if “the criteria have been established at a level sufficiently high enough to be appropriate at the 
freshmen through senior years and if faculty have been honest in the application of their own 
criteria” (King’s College, 1999b, p. 33). In addition, faculty in the major programs formally 
evaluated the sophomore-junior diagnostic projects and senior integrated assessments to 
determine the highest levels of performance, the mid-level, and those students who need 
improvement. The assessment results, including samples of actual student performances, were 
formally shared with faculty and administration. Project team members reviewed each other’s 
work to enhance inter-rater reliability based upon the use of common criteria. 



Advantages and Challenges Associated with a Competency-Based Educational System 

In the core general education curriculum, project teams frequently used common assessment 
activities in multiple sections of certain courses. This approach facilitated an extensive 
discussion of the assessment findings (King’s College, 1999b). Based upon the assessment 
results, a couple of faculty teams made key decisions about revising their goals, prompted 
discussion about teaching strategies among teams, and helped three additional teams determine 
whether their courses had been successful. For example, in the general education Civilization 
course, the project team decided to revise the pre- and post-assessments for this course since the 
multiple-choice tests did not reveal if students had met the higher order reasoning expectations 
for studying history on the college level. A new qualitative assessment activity was being pilot 
tested to determine if students expanded their knowledge of historical information, were able to 
correlate these facts with cause and effect, and possessed an understanding of historical 
significance (King’s College, 1999b). By contrast, the project teams who did not use common 
assessment activities reported that they did not discuss assessment findings. 

Most department chairpersons believed the sophomore-junior diagnostic project to be the most 
valuable component of the assessment model, since department faculty had useful information to 
determine the effectiveness of the first 2 years of education at King’s College (King’s College, 
1999b). This project also helped faculty to identify students who needed help at an early stage so 
that there was time to intervene effectively to help students overcome specific challenges. These 
plans particularly helped build students’ skills for their advanced courses since undergraduates 
were repeatedly working on these skills in multiple courses each year. Faculty believed that 
students effectively mastered key skills that were particularly important in the workplace. 

Through focus groups, students stated that they found course syllabi were more detailed and more 
directed toward student learning, and they liked knowing early the criteria faculty would use to 
judge their performance (Farmer, 1999b). In addition, students believed there were increased 
opportunities for them to have more meaningful interactions with faculty, and they had greater 
awareness of their sequential growth and development as learners. 

A major challenge associated with developing this type of system was that it took a considerable 
amount of time. King’s College did not attempt to change everything at one time. An 
incremental approach was taken whereby important changes were broken down into a series of 
discrete processes that occurred over time. In addition, the strategy of pilot-testing desired 
changes helped to encourage faculty to try these new ideas since they knew there would be 
opportunities for revisions (Farmer, 1988). 
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Recommendations for Other Colleges and Universities 

Other colleges and universities may be interested in adopting or tailoring the assessment program 
at King’s College to their own institution. Nearly all faculty agreed that it would be very difficult 
to adopt an identical form with the same definitions of liberal learning competencies at other 
colleges. However, the structure and process used at King’s College may be transferable to other 
institutions. For example, all faculty in major programs could be asked to identify the objectives 
for student learning, to indicate which courses will address these objectives and how they will be 
assessed, and to state very specific assessment criteria. In addition, faculty at other colleges could 
consider which particular strands of liberal learning (such as critical thinking or effective writing) 
should be emphasized in all majors and within certain upper level courses too. This does not 
mean that all courses should be designed to accomplish all liberal learning goals; rather, 
departmental faculty could make a commitment to focus on certain goals in numerous courses in 
their own major program. The benefit is that students continue to practice and build upon their 
own skills, frequently applying them to different contexts. 

In order for a model similar to King’s College systemwide change to be adopted, there were 
several key recommendations that faculty strongly advocated. First, faculty development was 
essential. At King’s College, the faculty development plan was more extensive than merely 
bringing an occasional speaker once a year. Faculty worked on one particular outcome rather 
than trying to do everything at one time. For example, there was a course offered during the 
entire semester about critical thinking, which faculty were strongly encouraged to attend. Getting 
faculty motivated to participate can be a challenge. Faculty believed the system at King’s 
College worked well, especially since incentives included getting release time from a course or 
being paid a stipend to develop and participate in faculty development during the summer 
months. These types of compensations helped faculty to get the necessary time and develop 
expertise to redesign their courses. Most faculty believed this type of system was critical to the 
success of this endeavor. 

Another key factor was strong leadership and support by the central administration. The academic 
vice president was often described as being very passionate about this competency-based work. 
He publicly thanked individuals and maintained open and frequent communications with 
colleagues. Professors emphasized that faculty ownership for this competency-based education 
system is very important, and that it exists at King’s College. However, they credited the 
academic vice president for providing the vision and guidance for conceptualizing and helping 
the implementation of these innovations to be successful. The leadership was clearly not 
authoritative. Conditions of trust and an environment open to experiments (often through pilot 
studies) was supported by the leadership. In addition, faculty highly valued the multiple 
opportunities to present their work on campus or at national conferences and to participate in 
faculty development seminars. 



Conclusion 

The King’s College faculty shifted their emphasis from instruction and teaching to a focus on 
producing learning. Faculty in the core and major programs worked together to articulate 
sophisticated plans that clearly document their intentions in terms of specific competencies for 
each particular course, strategies for assessing them, and the specific assessment criteria that will 
be used. 



O 

ERIC 



65 



78 



Faculty usually have ideas about what they want students to learn, but it may be rare for them to 
formally share their specific expectations with students. At King’s College, the “very act of 
writing such criteria has helped faculty to sharpen their understanding as well as to become 
clearer about their objectives for student learning in the classroom” (Farmer, 1991, p. 32). 
Students also knew as beginning freshmen that the faculty in their major programs would have 
clear expectations for their learning that built upon the liberal learning skills gained in the general 
education core curriculum. 

Most faculty at King’s College adopted the learning paradigm requiring faculty to change their 
role from being the authorities and sources of all right answers to being guides and facilitators to 
help students discover potential solutions to complex issues found in society today. The faculty 
were highly committed to continuously strengthen and refine their innovative curriculum as well 
as the assessment process. 
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Attachment A-l 



KING’S COLLEGE 

COMPETENCY GROWTH PLANS FOR ACCOUNTING MAJOR 1 

A. CRITICAL THINKING COMPETENCY. 

Students majoring in Accounting will be able to apply critical thinking skills in preparing 
and analyzing financial statements, preparing budgets, making managerial decisions, 
preparing audit programs, and writing audit reports. 

B. EFFECTIVE WRITING COMPETENCY. 

Students majoring in Accounting will be able to apply effective writing skills to the 
preparation and analysis of financial statements, audit programs, audit reports and tax 
opinion letters. 

C. EFFECTIVE ORAL COMMUNICATION COMPETENCY. 

Students majoring in Accounting will be able to make oral presentations that are clear, 
technically accurate, complete, properly organized and effectively delivered. 

D. LIBRARY AND INFORMATION TECHNOLOGY COMPETENCY. 

Students majoring in Accounting will be able to plan and implement search strategies so as 
to identify and use major reference tools appropriate to Accounting. 

E. COMPUTER COMPETENCY. 

Students majoring in Accounting will be able to use computer technology utilizing 
commercial accounting software, spreadsheets, word processing, computer supported 
presentations and appropriate research techniques including CD-ROM and the Internet. 

F. QUANTITATIVE REASONING COMPETENCY. 

Students majoring in Accounting will be able to make decisions utilizing proper 
quantitative methods appropriate to the subject matter of the decision. 



1 Material copied with permission of Donald Farmer, King’s College. 
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Material copied with permission of Donald Fanner, King’s College. 



KING’S COLLEGE 



COMPETENCY GROWTH PLAN IN CRITICAL THINKING 
FOR STUDENTS MAJORING IN ACCOUNTING 



FRESHMAN 



COMPETENCY 

DESCRIPTION 


STRATEGY 


ASSESSMENT CRITERIA 


The student will be able to: 

1 . Distinguish an argument 
from a set of claims that are 
not inferentially related. 

2. Distinguish the functions of 
language to express and 
influence meaning. 

3. Distinguish the kinds and 
purposes of definitions. 

4. Distinguish between validity 
and soundness, as they are 
related to deductive 
arguments and to evaluate 
the strength of inductive and 
rhetorical arguments. 

5. Recognize the common 
fallacies in everyday 
reasoning. 

6. Recognize and assess 
reasoning in various fields: 
business, law, science, the 
arts, etc. 

7. Present arguments effectively 
in oral and written form. 


Core 100: Critical Thinking 

In addition to taking three major 
tests and performing textbook 
exercises, the student will 
complete two major writing 
assignments. 

1 . A critical analysis and 
evaluation of an extended 
argument. 

2. An argumentative essay in 
which the student generates 
an argument on a 
controversial topic. 


1 . The student demonstrates 
recall and understanding of 
the pivotal concepts and 
processes of reasoning. 

2. The student identifies an 
argument and distinguishes 
support from conclusion. 

3. The student identifies 
language problems, such as 
ambiguity, vagueness, and 
emotionally loaded language. 

4. The student identifies crucial 
fallacies in arguments. 

5. The student summarizes and 
reconstructs an argument 
contained in an extended 
prose passage. 

6. The student draws 
appropriate inferences from 
given data. 

7. The student recognizes 
hidden assumptions and 
implied premises and 
conclusions of an argument. 

8. The student distinguishes 
subarguments from the main 
argument in a prose passage. 

9. The student separates a 
problem into discrete units 
and sets forth evidence in 
separate, meaningful 
categories. 
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KING’S COLLEGE 



FRESHMAN 



COMPETENCY 

DESCRIPTION 


STRATEGY 


ASSESSMENT CRITERIA 






10. The student uses the results 
of appropriate research 
(library, expert opinion, 
survey, poll, experiment, 
etc.) in the analysis, 
construction, and evaluation 
of arguments. 

1 1 . The student identifies and 
explains the reasoning 
process applied to various 
disciplines and demonstrates 
that process by constructing a 
strong argument in one of 
those fields, preferably his or 
her major discipline. 

12. The student recognizes and 
performs the basic functions 
of reasoning. 

13. The student chooses and 
defends an appropriate 
course of action from among 
a number of possible 
alternatives. 

14. The student relates an 
argument to broader issues 
and concerns. 

15. The student evaluates the 
acceptability of premises, 
their relevance to a 
conclusion and the adequacy 
of their support of that 
conclusion. 
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KING’S COLLEGE 



SOPHOMORE 



COMPETENCY 

DESCRIPTION 


STRATEGY 


ASSESSMENT CRITERIA 


The student will be able to: 

Define and explain the 
components and structure of the 
four financial statements: 

Balance Sheet, Income 
Statement, Statement of Cash 
Flows and Statement of Retained 
Earnings. 

Analyze the above statements, 
draw relevant conclusions about 
the individual or company, and 
communicate the results through 
written and oral techniques. 

Provide financial information 
that is useful in investment, 
credit and similar decisions along 
with information about a 
company’s resources, claims to 
those and changes in them. 


ACCT 260/270: Intermediate 
Accounting I and II 

• Preparation of homework 
assignment related to the four 
major financial statements. 

• Examinations consisting of 
problems and questions 
designed to encourage an 
understanding of the 
financial position and results 
of operations for a company. 

• Students present a financial 
analysis of a corporation. 


1 . The student defines, explains 
and distinguishes the content, 
organization, and purpose of 
the four major financial 
statements. 

2. The student reviews the 
structure and content of the 
four financial statements and 
communicates this analysis. 

3. The student calculates 
financial ratios designed to 
evaluate liquidity, leverage 
activity, profitability and 
market strength of a 
particular company. 

4. The student applies 
horizontal and vertical 
analyses of various balance 
sheets and income statements 
and draws proper 
conclusions. 

5. The student identifies, 
measures and summarizes 
the relevant data needed to 
properly record and present 
the results of operations and 
the financial position of the 
company. 

6. The student reflects upon 
their learning and discusses 
issues related to financial 
accounting with their fellow 
students as part of the 
Sophomore- Junior diagnostic 
project. 
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APPENDIX B 



Northwest Missouri State University 



Introduction 

In the spring 2000 semester, seven faculty were interviewed at Northwest Missouri State 
University (NMSU) by the Working Group consultant. Each individual interview during the site 
visit lasted between 50 to 60 minutes. Participants were mostly full-time faculty representing 
different ranks (full, associate, and assistant professors), and two individuals provided major 
leadership for assessment. Numerous documents were reviewed by the consultant, including the 
college catalogue and other relevant documents shared by the participants. The purpose of this 
case study was to explore the competency-based educational system at Northwest Missouri State 
University with a particular focus on the development of competencies and how they are 
assessed. 

There are several specific dimensions of the competency-based educational approach that will be 
illustrated in this case study. First, an overview of competencies in general education will be 
outlined as required by the state (for systemwide transfers) and then linked to NMSU. Second, 
the assessment work at the university as well as documentation requirements will be outlined. 
Finally, a statewide writing consortium will be highlighted. 

The Setting 

Northwest Missouri State University is a moderately selective, learner-centered regional 
institution offering a focused range of undergraduate and graduate degree programs (NMSU, 
1998). It is located in a rural community of approximately 10,000 individuals. Historically, the 
university serves 19 different northwest Missouri counties, emphasizing agriculture, business, and 
education programs. The university, accredited by the North Central Association of Colleges and 
Universities, offers bachelor’s, master’s, and specialist in education degrees as well as 1- and 
2-year certificate programs. 

The current student enrollment is approximately 6,200 students. Approximately 5,200 
undergraduates and 1,000 graduate students are working toward 98 undergraduate degrees, 26 
master’s degrees, 4 education specialist degrees, and a cooperative doctoral program in 
educational leadership. Ninety percent of the student body are undergraduates, and 50 percent 
live in on-campus housing. The median age is 19, and only 12 percent are married. Fifty-eight 
percent of these students are female, and 50 percent are first-generation college students. 

Northwest is committed to providing students with a strong general education core to prepare 
undergraduates for a dynamic world. The university is a national leader in applying information 
technology to the learning processes and in promoting continuous quality improvement to 
enhance performance across major functions and activities (NMSU, 1998). 
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Northwest was the only degree granting college or university selected for a site visit as part of the 
prestigious Malcolm Baldridge Quality Award Program. In November 1997, the school was 
honored for its extraordinary achievements in applying quality improvement principles to its 
operations with the Missouri Quality Award in Education. 

The university has embraced a set of core values that are outlined in the catalogue: 

• “High expectations are the starting point for quality; 

• Quality education is talent development; 

• Learning is an active, not passive, process; 

• Assessment must link process improvement to individual achievement; 

• Instruction should be learner-centered and holistic, challenging students to utilize 
levels of cognition and to develop physically, socially, and ethically; 

• The living/leaming environment must be aligned with the academic goals of the 
University; 

• An effective curriculum promotes sustained interaction and teamwork among 
students, faculty, and staff; 

• The nurturing, development, and empowerment of employees at all levels are critical 
to a quality living/leaming environment; and 

• Ethical behavior will be modeled and promoted by the University” (NMSU, 1998, 

p. 8). 



In 1987, the senior leadership (University President and Provost) initiated the “culture of quality 
program.” In 1992, Northwest continued to promote their quality program by following the 
Malcolm Baldrige National Quality Award criteria as a conceptual framework to discuss quality 
and as a template for planning and agenda setting (Hubbard, 1999). This was introduced as a 
master plan to revitalize undergraduate education. The culture of quality plan was developed by 
identifying 42 best practices and resulted in specific actions to improve processes. Northwest 
created its culture based on the educational research work of Alexander Astin’s definition of 
quality defined as talent development. Success is measured by the value that is added to students, 
faculty, staff, and the region. 




Identifying Important Outcomes: Influence of Statewide Policy 

The Coordinating Board for Higher Education (CBHE) in Missouri oversees higher education for 
the entire state. CBHE “fosters a public policy framework that is committed to the values of 
access, quality, and efficiency for the state’s higher education system. As Missouri continues to 
increase aspiration and performance levels for all students, it will require an educational system 
that is responsive to the needs of students for easy mobility across institutions” (Missouri General 
Education Steering Committee, 2000, p. 1). 



The Coordinating Board for Higher Education believes that each college and university in the 
state should be responsible for establishing and maintaining standards of expectations for all 
students completing its courses, programs, certificates, or degrees. However, CBHE values an 
effective and efficient transfer of credits between and among its institutions to create a high 
quality, statewide system of higher education. For this reason, the General Education Steering 
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Committee of CBHE has set forth statewide general education policies that directly affect and 
influence general education at Northwest Missouri State University. This policy articulates “a 
rationale for general education; defines the responsibilities of institutions, faculty, and students 
for general education; and promotes broad curricular goals and student competencies that should 
result from institutional general education programs” (Missouri General Education Steering 
Committee, 2000, p. 3). Through these policies, CBHE expects to facilitate student transfers to 
ensure the portability of general education credit among Missouri’s colleges and universities. 
The Missouri General Education Steering Committee views general education as the curricular 
foundation that “encourages students to acquire and use the intellectual tools, knowledge, and 
creative capabilities necessary to study the world as it is, as it has been understood, and as it 
might be imagined. It also furnishes them with skills that enable them to deepen that 
understanding and to communicate it to others. Through general education, the academy equips 
students for success in their specialized areas of study and for fulfilled lives as educated persons, 
as active citizens, and as effective contributors to their own prosperity and to the general welfare” 

(P- 3). 

Each institution is expected to ensure the transferability of general education credits among 
Missouri institutions. Faculty at each college or university specify and publish a 42-semester- 
hour block of general education credit that will be considered equivalent to corresponding blocks 
of credit at other public and signatory institutions in enabling students to achieve these general 
education goals and competencies (Missouri General Education Steering Committee, 2000, p. 4). 

CBHE defines goals as the “curricular intent of state policy regarding the academic skills and 
knowledge content of general education” (p. 4). Competencies consist of the illustrative 
state-level expectations for student performance in general education. Faculty (at each college or 
university) design a general education program that fits their mission and meets the state-level 
curricular goals. Faculty also must specify institution-level competencies that will follow from 
achieving these curricular goals and are in alignment with the suggested competencies created by 
CBHE. CBHE in turn has articulated state-level curricular goals and institution-level student 
competencies for general education in two categories: academic skills and knowledge. The 
specific skills include communicating, higher order thinking, managing information, and valuing. 
The knowledge areas include social and behavioral sciences, humanities and fine arts, 
mathematics, and natural sciences. Two examples from each category are outlined below. The 
statewide general education policy, including the types of examples listed below, was formally 
adopted on June 8, 2000, by the Missouri Coordinating Board for Higher Education. 



Two Examples of Skills Required by CBHE 1 
Communicating 

State-level goal: To develop students’ effective use of the English language and quantitative and 
other symbolic systems essential to their success in school and in the world. Students should be 
able to read and listen critically and to write and speak with thoughtfulness, clarity, coherence, 
and persuasiveness. 

Suggested competencies: Students will demonstrate the ability to: 

• Analyze and evaluate their own and others’ speaking and writing; 




'This material can be found on the Missouri Coordinating Board Web Site. See httoV/www. mocbhe.gov/acadafrs/gepolicv.htm 
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• Conceive of writing as a recursive process that involves many strategies, including 
generating material, evaluating sources when used, drafting, revising, and editing; 

• Make formal written and oral presentations employing correct diction, syntax, usage, 
grammar, and mechanics; 

• Focus on a purpose (for example, explaining, problem solving, argument) and vary 
approaches to writing and speaking based on that purpose; 

• Respond to the needs of different venues and audiences and choose words for 
appropriateness and effect; 

• Communicate effectively in groups by listening, reflecting, and responding 
appropriately and in context; and 

• Use mathematical, statistical models, standard quantitative symbols, and various 
graphical tactics to present information with clarity, accuracy, and precision. 

Higher order thinking 

State-level goal: To develop students’ ability to distinguish among opinions, facts, and 
inferences; to identify underlying or implicit assumptions; to make informed judgments; and to 
solve problems by applying evaluative standards. 

Suggested competencies: Students will demonstrate the ability to: 

« Recognize the problematic elements of presentations of information and arguments and 
to formulate diagnostic questions for resolving issues and solving problems; 

• Use linguistic, mathematical, or other symbolic approaches to describe problems, 
identify alternative solutions, and make reasoned choices among those solutions; 

• Analyze and synthesize information from a variety of sources and apply the results to 
resolving complex situations and problems; 

• Defend conclusions using relevant evidence and reasoned argument; and 

• Reflect on and evaluate their critical thinking processes. 



Two Examples of Knowledge Areas Required by CBHE 2 
Social and behavioral sciences 

State-level goal: To develop students’ understanding of themselves and the world around them 
through the study of content and the processes used by historians and social scientists to discover, 
describe, explain, and predict human behavior and social systems. Students must understand the 
diversities and complexities of the cultural and social world, past and present, and come to an 
informed sense of self and others. 



Suggested competencies: Students will demonstrate the ability to: 

• Explain social institutions, structures, and processes across a range of historical periods 
and cultures; 




2 Material copied with permission of David Oehler, Missouri Coordinating Board of Higher Education. 
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• Develop and communicate hypothetical explanations for individual human behavior 
within the large-scale historical and social context; 

• Draw on history and the social sciences to evaluate contemporary problems; 

• Articulate the interconnectedness of people and places around the globe; 

• Describe and analytically compare social, cultural, and historical settings and processes 
other than one’s own; and 

• Describe and explain the constitutions of the United States and Missouri. 
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Humanities and fine arts 

State-level goal: To develop students’ understanding of the ways in which humans have 
addressed their condition through imaginative work in the humanities and fine arts; to deepen 
their understanding of how that imaginative process is informed and limited by social, cultural, 
linguistic, and historical circumstances, and to appreciate the world of the creative imagination as 
a form of knowledge. 

Suggested competencies: Students will demonstrate the ability to: 

• Describe the scope and variety of works in the humanities and fine arts (for example, 
fine and performing arts, literature, and speculative thought); 

• Explain the historical, cultural, and social contexts of the humanities and fine arts; 

• Identify the aesthetic standards used to make critical judgments in various artistic fields; 

• Develop a plausible understanding of the differences and relationships between formal 
and popular culture; and 

• Articulate a response based upon aesthetic standards to observance of works in the 
humanities and fine arts; 

All Missouri public colleges and universities, as well as independent or proprietary institutions 
that are “signatory to the statewide credit transfer policy, are expected to develop and post (both 
on the CBHE Web Site and their own institution’s Web Site) the curricular design and assessment 
plan indicating how that institution plans to implement and assess general education” (Missouri 
General Education Steering Committee, 2000). 

A general education transfer policy has been in place since 1987. A comparison of the “old” 
policy adopted in 1987 with the proposed current version (that will most likely be implemented) 
reveals some changes in the following areas. Originally, in 1987, only 39 credit hours were 
required in the lower division core curriculum. Now, 42 credit hours are proposed. In the original 
policy, there was no explicit statement of rationale for general education. The current proposal 
articulates an explicit rationale for general education. In the 1987 policy, programs could be 
distributional, topical, or thematic. However, there were no explicit goals or objectives of general 
education. The current proposal has shifted toward giving faculty flexibility to design programs, 
but they must be aligned with eight explicit and prescribed state-level general education goals. 
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“While each goal is accompanied by a list of specific student competencies, these competencies 
are for illustrative purposes. Faculty may substitute institution-level competencies for each goal 
that fits the ethos and mission of their respective institution” (Missouri General Education 
Steering Committee, 2000). Finally, in 1987 faculty of each institution could control their general 
education program. The current policy allows faculty to design their own general education 
curricula, but they must designate a 42-semester-hour block of transferable general education 
credit designed to meet state goals. 



Identifying Important Outcomes: Northwest Missouri State’s Approach 

Currently, all employees at Northwest Missouri State embrace the student-centered culture of the 
quality framework. In the early 1980s, faculty and administrators wanted to better address the 
needs of students and other stakeholders. Therefore, in 1987, a master plan was introduced to 
revitalize undergraduate education. This plan was developed by identifying the 42 best practices 
and resulted in specific actions to improve processes at NMSU. The University now has a shared 
set of values that mainly focus on exceeding student expectations and striving for continuous 
learning and improvement at all institutional levels. 

Students, including those who have completed programs and those currently enrolled, are the 
main customers. Stakeholders include employers, professional schools, graduate schools, and 
other organizations that may benefit from NMSU alumni. 

The President of NMSU surveyed Chief Executive Officers of major national corporations 
through a Delphi process to determine the knowledge and skills that undergraduates need to be 
effective in the workplace. The results from this survey were incorporated into the specific 
university Key Quality Indicators (KQIs) or competencies outlined below: 

1. Communication — Goals are to generate researched writings that develop and 
organize a valuable central idea; use writing processes successfully to invent, plan, 
draft, revise, and edit; comprehend presentational and interpersonal messages; use 
critical and empathetic listening; prepare and perform informative and persuasive 
speeches; understand and appreciate literature and its study; and apply basic literary 
methods in readings, discussions, and writings. 

2. Problem solving, critical and creative thinking — Goals are to interpret, evaluate, 
and understand scientific information and processes and their applications; 
understand the historical and modem relationship among science, society, and life 
experiences; apply appropriate analytic geometric and/or statistical skills; use 
formula, data analysis, information, events, and/or graphic representation to make 
allocation decisions or solve other problems with numerical answers; access relevant 
data needed to solve problems; recognize/create strong arguments/theses/narratives; 
and use abstract paradigms/models to analyze, interpret, and evaluate data / 
information. 

3. Computer — Goals are to understand how computers and emerging, related 
technologies impact society; and to use computers and related technologies in their 
disciplines and everyday life. 

4. Self-directed learning — Goal is to empower student to become a lifelong learner. 
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5. Competence in a discipline — Goals are to gain in-depth study of a major field of 
investigation while gaining the competencies required of all graduates, and to 
develop an understanding of the discipline and how it relates to society. 

6. Personal and social — Goals are to relate to and interact effectively with others and 
be able to adapt to a variety of social and work environments; manage their personal 
affairs and contribute positively and productively to society; and explore their 
personal values, talents, interests, and aspirations with respect to lifelong goals. 

7. Multicultural — Goals are to have an awareness of cultural diversity; develop the 
ability to interact with various cultures; and develop an appreciation for various 
manifestations of culture. 

8. Cultural enrichment — Goals are to have an appreciation and understanding of the 
arts including the artist and the creative process as they affect and reflect society; 
have a desire to participate in aesthetic experiences; learn and evaluate literature from 
student’s own culture; and learn and evaluate literature from other cultures. 



In addition, the university seeks to foster a culture of commitment to service with the following 
attributes: 
























Treating individuals with respect, fairness, and honesty; 

Performing tasks with competence and skill; 

Communicating clearly and courteously the services provided; 

Listening actively to requests, comments, and concerns; 

Being flexible and open to new ideas; 

Providing what is agreed upon to deliver in a timely manner; and 
Maintaining a safe and orderly, healthy, well-functioning, and attractive campus. 



Formal feedback is also gathered from numerous advisory boards as well as from national studies 
of employer expectations. Many departments have their own advisory councils that serve to 
validate the KQIs. This validation is important since it serves as a mechanism to address the 
requirements and expectations that employers will have for new graduates in the work place. 

The faculty are currently reviewing their own general education program to identify how their 
existing curricula fit with and actually map onto the requirements outlined above by CBHE. The 
faculty will specifically be examining whether there are any gaps that need to be addressed by 
their own general education program. 

Responsibility for tracking and implementation of the key quality indicators is conducted by the 
General Education Continuous Quality Improvement Advisory Group, which has been formalized 
as the General Education Subcommittee of the Faculty Senate’s Curriculum Committee. 
Academic departments are responsible for implementing and keeping track of their own KQIs. 
However, the eight competencies outlined above must cut across both general education and 
academic programs. 
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Assessment Strategies at Northwest Missouri State University 

The overall assessment plan is coordinated by the Director of Assessment, Information, and 
Analysis. This office consists of two full-time staff and one graduate assistant. In addition to this 
staffing pattern, the university spends about $55,000 on assessment-related activities. 

There are numerous assessment strategies that faculty use to determine if students are mastering 
the expected competencies. These assessments include both direct and indirect measures. Some 
measures are locally developed instruments, while others are standardized tests. Some 
assessments are common for all students, while faculty may design their own individual 
assessments for their particular courses. In this section, a brief review of major assessments will 
be presented. 



Commercially Developed Instruments 

Undergraduates are required to take the Academic Profile (the short form) published by the 
Educational Testing Service. This instrument assesses college-level reading, critical thinking, and 
writing mechanics within the context of material from the humanities and the social and natural 
sciences. 

The university gets one report of the entire group’s results. Currently, the assessment director is 
experimenting with breaking down the results into subgroups by academic departments. They are 
beginning this new method of examining results and do not have information yet on the 
usefulness of this approach, but the overall group scores for the Academic Profile have increased 
somewhat from spring 1998 to fall 1999. All seniors must take this test in order to register for 
their classes. However, there are no consequences for students who perform poorly on this 
instrument. The institution’s leaders would like all students to score above the 50th percentile. 
No specific changes have been made based upon the test results. 

A new version of the Academic Profile is available at www.ets.org.hea . With this new version, 
NMSU is able to provide students and advisers with undergraduate’s individual results. 

Initially the Coordinating Board of Higher Education would only fund assessment results from 
commercially developed tests. Now, CBHE will provide some funding for locally developed 
approaches. However, CBHE still provides more money to institutions that use commercially 
developed tests. Institutions receive a certain amount of funding depending upon how well their 
students perform on these tests. Therefore, in reality, institutions may be more motivated to use 
such tests since there are higher levels of resources offered for these particular types of 
instruments. 
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Writing Assessment 



In the general education program, there is a sophisticated example of a common assessment 
across all writing courses required for undergraduates. Faculty have designed an end-of-core 
writing assessment that students take at the end of the second required writing course. Students 
are given a series of short, related readings (often from newspapers or magazines) about a 
controversial topic and then asked to respond to specific prompts. One example is listed below. 



Prompts and Topics 3 

Using and citing evidence from at least two of the readings (you may use more than two), along 
with your own experience and common knowledge, develop a well organized essay on one of the 
following topics: 

Topic A: What causes “hate crimes 11 in this country and what should be done to prevent them? 

Topic B: Do you agree or disagree that hate crime legislation would be effective? 

Remember to write an introductory thesis paragraph, a body of paragraphs with detailed evidence 
to support your thesis and a conclusion. Cite your sources in MLA style. 

The end-of-core writing assessment rubric consists of the following dimensions outlined below . 4 
Score of 5 = Excellent 

Organization: Impressive introduction, thesis, conclusion, essay plan, sophisticated transitions. 

Development: Substantial evidence, and adapts sources easily to defend thesis. 

Language: Skillful attention to sentence structure and style, no grammatical errors that inhibit 
clarity, few usage, punctuation, or spelling errors. 

Documentation: Careful attention to academic conventions for citation and proper use of 
sources, avoiding all plagiarism. 

Score of 3 = Adequate 

Organization: Functional thesis and introduction; apparent organization with few transitions; 
some logic problems. 

Development: Sufficient examples and detail to support a thesis that makes a valuable statement; 
some use of sources, some lack of evidence. 

Language: Functional sentence structure; some cliches; some grammatical, mechanical or usage 
errors. 

Documentation: Adequate attention to academic conventions for citation and proper use of 
sources, avoiding plagiarism. 




3,4 Material copied with permission of David Oehler, Northwest Missouri State University. 
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Score of 1 = Inadequate 

Organization: None, or missing connections between thesis and paragraphs; severe logic 
problems. 

Development: Few, if any details; use only one or no sources; thesis based on generalizations or 
prejudice; undue reliance on stacked quotes. 

Language: Incoherent sentence structure and word choice, frequent grammatical, mechanical, 
usage errors. 

Documentation: Lack of attention to academic conventions for citation and proper use of 
sources, perhaps committing plagiarism. 

Students receive information about the specific criteria that is usually outlined in the course 
syllabus. In addition, each course syllabus in the writing course sequence also articulates and 
reinforces the same competencies. These competencies include: 

• The practice of effective writing methods; 

• The integration of reading and writing methods productively; 

• The application of appropriate rhetorical and communication strategies in their 
writings; 

• The development and organization of thorough writings centered in valuable ideas; 

• The generation of documented academic essays that display a fully literate style and 
correct usage; and 

• The possession of the knowledge, skills, and attitudes of competent collegiate 
writers. 

These writing competencies and course sequence are designed to fulfill the communication and 
critical/creative thinking Key Quality Indicators of the university’s curriculum. 

Undergraduates who receive an inadequate score on the end-of-core assessment must take the 
assessment again. If the second assessment also is inadequate, then the students have two options: 

1. Put together a portfolio before the end of the semester. If both the second 
assessment and the portfolio are inadequate, then the student must retake the 
English two-course sequence again. 

2. Take a delayed grade if the second assessment also receives the inadequate score. 
During the next semester, the student must retake the end-of-core assessment a third 
time. A student may put together a portfolio by the end of that semester if the third 
assessment is also inadequate. If both the assessment and portfolio receive an 
inadequate, then the student will need to repeat the English two-course sequence. 
The portfolio must include a letter introducing the contents, a research paper from 
one of the composition courses (including both a revised and graded copy), and an 
essay of the student’s choice (including both a revised and graded copy). 
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The writing assessments and the portfolios are read and assessed by two members of the English 
Department with a third reader if the initial two readers are not in agreement. Faculty members 
are trained in how to assess student writing. Faculty initially reviewed a sample of student work 
to determine how close their individual ratings are to each other’s scores. If there are wide 
discrepancies, they discuss their reasons for scoring student writing in a certain way. Once a 
fairly high level of consistency is reached, two faculty members read each end-of-core writing 
assessment. If there are major disagreements between the two readers, then a third reader scores 
the essay, which brings closure to the assessment. All faculty readers are paid extra compensation 
for the time involved in reading these essays. 

In 1996, about 90 percent of undergraduates passed the test the first time; more than 99 percent 
passed after completing the test sequence. About half of the failures were typically for plagiarism. 
In spring 1997, 82 percent of students passed the end of core assessment the first time, and 99 
percent passed after completing the test sequence. 



Writing Assessment Strategies Across Colleges and Universities in Missouri 

The Missouri Colloquium on Writing Assessment (MCWA) is a statewide organization that exists 
to support faculty members (frequently professors of composition and literature) in all 2- and 
4-year public and private institutions in Missouri. This group believes that assessments of writing 
should primarily be used to improve instruction and “empower students to become successful 
writers” (MCWA, 1999, p. 2). Typically, the Colloquium convenes a statewide meeting once per 
year where colleagues review and discuss current issues in writing assessments. The group also 
surveys faculty about writing instruction and assessment trends and publishes a newsletter for 
their members. 

In the fall 1999 Colloquium, faculty reported some major assessment challenges. One institution 
reported that “results were purposefully not reported locally because the assessment instrument 
was externally imposed and not matched to local instructional objectives” (MCWA, 1999, p. 3). 
Other college faculty had problems with getting their colleagues to conduct writing assessments 
and cited the following reasons: “(1) they do not want to be assessed because they do not want to 
be accountable or feel that they are above that kind of accountability; (2) they want to assess, but 
do not believe that the instruments/data provide meaningful results; (3) they fear that assessment 
programs are fronts for secret administrative faculty-surveillance agendas; and (4) they are simply 
weary from the work and expense that assessment (especially large-scale assessment) demands” 
(MCWA, 1999, p. 3). 
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Beginning in 1989, the MCWA has conducted a yearly writing survey at all institutions within the 
state (although mostly public institutions respond). Highlights of the results from the 1999-2000 
do not indicate the specific competencies that students should master. The results indicate how 
students are assessed, but do not indicate how well students perform or how the results are used to 
make changes or improvements. However, the findings illustrate that “Missouri institutions are at 
the forefront in the movement to use multiple measures — portfolios, questionnaires, writing 
conferences, impromptu essays, essays that allow drafting time to judge the success of students 
and writing programs” (MCWA, 1999, p. 7). Sixteen colleges and universities implement 
programmatic writing assessments rather than depending upon only nationally normed tests to 
assess the writing competencies of undergraduates. Twelve of the 27 institutions have a director 
or coordinator of writing assessment. Faculty in these leadership roles typically have teaching 
load reductions ranging from 3 hours to 12 hours per year. Readers of students’ assessment work 
normally get paid for their time. The compensation ranges from $10 to $50 per hour, while the 
average pay is $22.30 per hour. It was beyond the scope of this case study to identify why there 
are wide ranges in terms of support. However, it is important to note that faculty leaders were 
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released from courses so that they could focus their energies on the writing assessments. In 
addition, all readers were given extra compensation for their time. 



Documenting Student Achievement 

Faculty at Northwest Missouri State University follow a seven-step planning process that applies 
to general education as well as all major programs. This “process is designed to bring a common, 
more rigorous planning language to all units of the University and to enhance departmental 
communication and understanding about programs. Ultimately, it should lead to a University that 
is more market driven, coherent and efficient in its overall design and operations” (Oehler and 
Knapp, 1996, p. 1). The seven-step planning process is designed to help university faculty 
accomplish the following: 

• “Define a set(s) of major objectives or key quality indicators, stated in terms of the 
customers’ needs, that serve to focus the team’s efforts on a shared set of aims; 

• Review and strengthen the design, content and delivery of the teams’ 
curriculum/operations so that they more closely match the needs of their customers 
and better integrate and coordinate the work of the team members; 

• Define a set of outcomes measure that accurately gauge the extent to which the 
customers’ needs are met and that can be used to track progress toward department 
team, college and university performance improvement goals; and 

• With a single process meet many of the external process improvement requirements 
including the Campus Based Review, Funding for Results, Regional and Specialized 
Accreditation and the Baldridge Assessment Framework” (Oehler and Knapp, 1996, 

p. 1). 
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Currently, the seven-step planning process (as applied to both general education and the academic 
programs) consists of initially the articulation of the general competency followed by seven more 
specific goals, and then critical success factors (see attachment B-l for the Communication 
Competencies example). The critical success factors are even more descriptive attributes 
associated with each particular goal. Each goal has at least two critical success factors. Then the 
deployment strategies are identified. Faculty formally identify the specific courses in the general 
education program and the majors where students are expected to master the goals and critical 
success factors. The individuals or committees assigned primarily responsibility are identified. 
Then the particular assessment measures are articulated for the goals. The assessment frequency 
is identified, as is the intended audience for sharing the results and the intended usage. Next a 
summary of results is described, as well as trends over time (if they are available), and 
competitive comparisons are cited when possible. This seven-step planning process is followed to 
document each of the 10 competencies both in general education and the academic majors. It 
provides a strong map that guides the development, implementation, and assessment of 
competencies across the undergraduate curriculum. 

All faculty are required to use this process for their academic programs and for general education. 
This process is completed on an annual basis, and the individual unit has a formal conversation 
with the appropriate dean about its accomplishments, how it needs to change to address new 
challenges and opportunities, and strategies for continually improving. Once each 5 years, these 
plans are formally reviewed by the President, Board of Regents, the Faculty Senate Curriculum 
Committee, the Faculty Senate Assessment Committee, the Graduate Dean (for departments with 
graduate programs), and the Coordinating Board for Higher Education. This seven-step planning 
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process is embedded within a larger planning process outlined below. The specific format and 
points to be addressed include the following: 

Section 1: What factors are influencing your unit’s current and future outlook? 

A. What is your unit’s mission? 

B. What external and internal factors bear on your unit’s outlook for the future? 

C. Give the factors above, should your mission change? How? 

D. What is your unit’s five-year vision? How should this vision change given the 
factors above and your thinking about the unit’s mission? 

E. How does this vision enable your unit to contribute better to the University’s 
achievement of its mission? 

Section 2: What is the current status of your unit? 

A. Provide your unit’s Seven-Step Plan. 

B. What is your analysis and interpretation of the Seven-Step Planning Process 
results? 

C. With regard to performance targets set jointly with University leadership as part of 
your unit’s Seven-Step Planning? 

D. With regard to workload objectives established for your unit? 

E. What is your unit’s strategy for developing its faculty and staff and what resources 
has it invested in the strategy’s achievement this year? 

F. What are the key accomplishments of your faculty, staff, and students this year? 

Section 3: What future directions have you set for your unit? 

A. What are your performance targets, jointly established with University leaders, for 
the coming year? For the next five years? 

B. What is your action plan for achieving these targets, and what resources 
will be needed? 

C. How do these targets contribute to the achievement of University-wide Strategic 
Initiatives and directions? 

Students are the main customers (including those who have completed programs and those 
currently enrolled). Stakeholders include employers, professional schools, graduate schools, and 
other organizations that may benefit from Northwest Missouri State University alumni. 

One major future initiative at NMSU will be the development of an electronic portfolio. The 
vision is that all assessment results for individual students, including assessments of their 
out-of-class activities, will be posted electronically. 
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Data Ramifications 



Faculty use the Academic Profile to assess the general education competencies of their 
undergraduates. Since the test is commercially developed by the Educational Testing Service and 
used for many years by different colleges and universities, there is strong record of reliability and 
validity. The greatest challenge is how to use the results to make improvements at individual 
institutions. Currently, NMSU staff and faculty mainly review the aggregate scores of their entire 
group of students and do not know how subgroups of students are performing. They review the 
aggregate scores to determine if there are improvements or changes over time. Scores have been 
fairly steady, with an increase from 1998 to 1999. The assessment director is exploring dividing 
the data into subgroups by academic departments, but this is a relatively new idea so there are no 
results about its usefulness. 

The locally developed end-of-core writing assessment has an increased reliability since all readers 
are formally trained about how to evaluate student work. Through this formal process, faculty 
better understand the scoring criteria and reach higher levels of consistency in their judgments 
about student work. Faculty also believe that a major strength of this assessment is that students 
must address a real-world issue or problem. Validity is somewhat enhanced since students must 
read about a particular issue from multiple sources of articles rather than relying only on one 
document. 

In the future, some faculty expect that NMSU will use more locally developed assessment 
methods since these instruments are more closely aligned with the professors’ expectations for 
student learning. As the university moves in this direction, faculty realize that they will need time 
to pilot instruments and assistance to determine the reliability and validity of these methods. 



Advantages and Challenges Associated With a Competency-Based Educational System 

Faculty emphasized that the development of a competency-based model is a very complex 
process that requires a major commitment of time. Some faculty believed that at least 1 year is 
needed when beginning to get teams of individuals together and dialoguing so that they feel real 
ownership of the outcomes. A comprehensive plan is needed rather than a piecemeal approach. 
Professors believed that they had a good history of support for faculty development from NMSU 
in terms of release time for their work and summer stipends. 

The greatest challenge according to the faculty is the amount of time and energy required for the 
documentation (Seven-Step Planning Process). Such a formal process requires high levels of 
support, especially from department chairs, to conceptualize, review, and implement their plans. 
Another level of complexity is added by the specific requirements mandated by the Missouri 
Higher Education Coordinating Board. Faculty must insure that the general education program 
adheres to the policies of CBHE. However, the planning process is very useful since it helps 
faculty to think about the needs of their own students and the larger needs of the entire state. 



Recommendations for Other Colleges and Universities 
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Faculty stressed the importance of providing release time during the semester so that instructors 
can seriously work on their course and program revisions. Stipends are critical in the summer for 
faculty since they typically have a bit more time to reflect upon their designs and curriculum 
changes. Faculty development is also crucial since professors often need some formal time to 
work with colleagues and learn about new ideas. 
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At NMSU, faculty usually have two full professional development days at the beginning of a 
semester before classes begin. They reported that these days were very important to their 
successful plans and implementation. 

Faculty also emphasized that it is important for senior administrators and academic leaders to 
build a culture open to change. This culture is also focused on the improvement of education and 
the establishment of regular feedback mechanisms to better meet the needs of their customers — 
the students. 



Conclusion 

The senior leadership at Northwest Missouri State University have succeeded in building and 
sustaining an institutional culture of quality. The use of key quality indicators helps the university 
to link broader goals for general education with competencies across different majors. Since the 
same quality indicators apply across programs and general education, there is also greater 
coherence among the learning experiences for undergraduate students. In addition, students have 
opportunities to build upon their skills and refine them through multiple courses across the 
curriculum. A major strength of this approach is the fact that the assessment plans are fully 
embedded in a larger strategic planning process required of all units and programs. 

Faculty have shifted their primary emphasis from an instruction model (based on what faculty 
want to teach) to a more learner-centered approach. Through multiple methods, faculty determine 
specifically what undergraduates need to be successful in the real world and then build learning 
experiences to address those needs. Finally, faculty formally assess student learning to determine 
if students are mastering the important outcomes. 
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Material copied with permission of David Oehler, Northwest Missouri State University. 



GENERAL EDUCATION: CORE AND LIBERAL STUDIES 
Seven-Step Process Matrix (Continued) 
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GENERAL EDUCATION: CORE AND LIBERAL STUDIES 
Seven-Step Process Matrix (continued) 
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GENERAL EDUCATION: CORE AND LIBERAL STUDIES 
Seven-Step Process Matrix (continued) 
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GENERAL EDUCATION: CORE AND LIBERAL STUDIES 
Seven-Step Process Matrix (continued) 
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Notes: AOS: Alumni Outcomes Survey; sample of Northwest alumni 3 years after graduation. 

COS: College Outcomes survey; 50 percent of the first semester sophomore class. 

CSEQ: College Students’ Experience Questionnaire; 50 percent of the first semester sophomore class. 



APPENDIX C 



Sinclair Community College 



Introduction 

The purpose of this case study was to explore the competency-based educational system at Sinclair 
Community College with a particular focus on the development of competencies and how they are 
assessed. 

In the fall 1999 semester, four faculty who worked at Sinclair Community College were interviewed by 
the Working Group consultant. Each individual interview lasted between 40 to 50 minutes. 
Participants were full-time faculty representing different ranks (full, associate, and assistant 
professors). Documents reviewed by the consultant, including the college catalogue and selected 
reports about Sinclair Community College, described their assessment plans and results. 

The Setting 

Sinclair Community College is a comprehensive 2-year community college offering a diverse range of 
about 1,500 university-parallel, technical, and career courses to a student body of approximately 18,000 
students. Sinclair is the largest single-campus community college in Ohio and one of the 20 largest in 
the United States. It is accredited by the North Central Association of Colleges and Schools and is 
located in an urban inner city. 

Sinclair Community College has six degree-granting academic divisions, including Allied Health, 
Business Technologies, Engineering and Industrial Technologies, Extended Learning and Human 
Services, Fine and Performing Arts, and Liberal Arts and Sciences. It is an open-door institution and 
admits all applicants into the college. 



Identifying Important Outcomes 

In the mid-1980s, Sinclair Community College administrators and faculty began contemplating the 
possibility of guaranteeing the entry-level skills of their graduates. However, they realized that before 
they could make such a guarantee, they had to clearly articulate student learning outcomes and 
determine if they were meeting employers’ needs. That was the first step toward defining 
competencies and developing an assessment plan at Sinclair. Faculty and administrators formally 
examined their program outcomes and identified both formative and summative assessment needs. In 
1992, Sinclair decided to ultimately adopt the guarantee of entry-level skills for their graduates. If 
graduates were not prepared for their entry-level jobs in certain areas, they could take up to nine credits 
of additional coursework at no additional charge. However, employers must certify in writing the 
specific job skills that need to be strengthened. This guarantee only applied to graduates employed on 
a full-time basis directly related to the area of program concentration as certified by the Vice-President 
for Instruction (Sinclair Community College, 1999). A similar guarantee was in place for students who 
transferred to another institution. 

Some faculty believed that there was an important need to identify and assess student outcomes since 
there were external requirements from the accreditation mandates by the North Central Association of 
Colleges and Schools. However, there was no formal state mandate in Ohio to implement assessment. 



Another reason for the focus on competencies and their assessment was that Sinclair embraced the 
principles of the “learning college” and wanted to move toward more active or collaborative 
experiences for their students. The learning college is based on six key principles (O’Banion, 1999, 
p. 5) and seeks to: 

• Create substantive change in individual learners; 

• Engage learners in the learning process as full partners who must assume primary 
responsibility for their own choices; 

• Create and offer as many options for learning as possible; 

• Assist learners to form and participate in collaborative learning activities; 

• Define the roles of learning facilitators in response to the needs of learners; and 

• Document improved and expanded learning of students. 



Many institutions have provided data about institutional effectiveness such as rates of graduation, 
persistence, or employment for selected alumni. These types of data are very important to share, 
especially with external stakeholders. However, the learning college concept as proposed by Terry 
O’Banion (1995) places an emphasis on the widespread engagement of faculty and staff in key 
discussions about the specific definitions of learning. Such discussions and decisions about what 
specific competencies comprise learning are important to examine as faculty consider the design of 
learning experiences. Faculty at Sinclair Community College believed that these types of substantial 
changes would lead to better retention of material across courses and higher levels of learning for their 
undergraduate students. 

At Sinclair, general education was defined as “supporting individuals in the quest to become whole, 
complete persons by encouraging the development in areas such as thought, communication, values, 
creativity feeling, adaptability and awareness” (Sinclair Community College, 1999, p. 54). The 
specific outcomes for general education are outlined in Table 1. Faculty identified which particular 
courses were designed to fulfill the competencies. 

“This general education program was designed to provide foundation skills necessary for successful 
living in the ever-changing present and future global environment” (Sinclair Community College, 
1999, p. 54). General education was also structured to provide a sense of community and foster 
collaborations. Students must complete 20 quarter hours of courses in communication, English, social 
science, humanities, mathematics, and computer literacy. This curriculum was primarily a set of 
distribution requirements where students had choices about which particular courses they should take. 
In addition to gaining educational breath, these courses supported the development of three major 
“across-the-curriculum” competencies, including communication, thinking, and values/citizenship/ 
community. 

Each academic program faculty group defined learning outcomes that indicated expectations for 
student achievement in their particular majors. In effect, these competencies served to define the 
specific knowledge and technical skills that college students should be able to demonstrate. The 
outcomes for each program are published in Continuous Improvement Through the Assessment of 
Student Learning Outcomes: A Work in Progress (Sinclair Community College, 1998). Faculty 
identified the specific courses that are targeted for preparing students to achieve each particular 
competency in the program. They usually also articulated the specific outcomes related to general 
education that are built upon in major courses. 
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Table 1. — Learning outcomes for general education 



Learning Outcomes 

1 . Communicate the significance of facts, 
ideas in spoken and 

written English that is clear, 
precise, and logical. 

2. Demonstrate a problem-solving capability 

through analysis and synthesis. 

3. Recognize ways in which a scientific 
approach can be used to formulate 
an understanding of the observable 
world. 

/ 

4. Recognize and describe interactions and 

institutions that characterize the 
the individual and society. 

5. Appraise the values and character of both 
Western and non-Westem cultures. 



6. Recognize and appraise various forms 
in which human creative efforts are 
expressed. 

7. Demonstrate academic proficiency 
comparable to students completing 
the second year of a baccalaureate 
degree program. 



Related Courses 

All English and Communication concepts, 
and courses 



All Math courses and all Science 
courses; Philosophy 207 

All Science and Social Science 
courses 



All Social Science courses 



Arts, History, Philosophy, 
Humanities, Foreign Language, and 
Social Science courses 

Literature, Art, Music, Dance, and 
Theatre courses 



All courses 



Assessment Strategies 

As a result of the open-door admissions policy at Sinclair, the abilities of entering groups of students 
varied substantially. Therefore, the college measured the entry-level competencies of students through 
COMPASS (Computer Adaptive Placement Testing), an assessment developed by the American 
College Testing company to ensure reliability and validity. COMPASS is a battery of assessment 
instruments designed specifically for use by community, technical, and junior colleges. The 
instruments assess reading skills, language usage, numerical skills, elementary algebra, intermediate 
algebra, and college algebra. Some programs used additional instruments for diagnostic and/or 
placement purposes. For example, the English department mandated that all students complete a 
writing sample on the first day of class to verify their placement based on COMPASS results. 
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It is frequently challenging to make decisions about assessments of general education. Sinclair initially 
focused on writing and communication. The specific competencies for writing skills are outlined in a 
checklist form (see attachment C-l). Some professors shared this list with their students and instructed 
them that it would be the basis for evaluating writing assignments. The checklist consisted of five 
major categories (with specific competencies defining each category), including assignment profile, 
content, organization, style, and mechanics. The rating scale indicated an evaluation of mastery of 
each writing skill by indicating an absolute yes or no or not applicable. A second oral communication 
checklist (see attachment C-l) was developed. Faculty followed a DACUM (Developing a 
Curriculum) process to identify important communication skills that faculty members and students 
thought were important. DACUM typically includes a panel of appropriate experts to identify general 
areas of competence and then specific competencies for each particular area. Through the DACUM 
process at Sinclair, participants categorized the important speech communication skills into the areas of 
public speaking, small group communication, interpersonal communication, and listening. Each 
particular communication skill area was then further defined by a range of 7 to 13 specific competency 
statements. The rating scale for the speech communications checklist consisted of a five-point range 
from totally absent to excellent. There were some early adopters, but the majority of faculty were not 
using it yet. 

These forms were used by faculty to evaluate student mastery of specific skills. Students themselves or 
their peers may also assess each other. The vision was that eventually these checklists would be used 
in the majority of courses since these skills were supposed to be developed in all disciplines. 
Additional checklists were being developed to define competencies in values/citizenship/community 
and in thinking. 

The Cornell Critical Thinking Test was piloted recently for students taking general education courses in 
sociology and psychology. Another pilot will be conducted for graduates of the Liberal Arts and 
Science degree programs. No formal decision was made to adopt this particular tool to assess general 
education outcomes. Other alternatives that were being considered included using portfolios or 
developing a freshman year experience introductory course where assessments could occur and be 
compared with results from a new capstone course at the end of the program. 

Each academic department at Sinclair developed or refined multiple methods to assess students’ 
achievement in specific learning outcomes within their major. Final development of each assessment 
plan was completed in June 1993. These plans were open to revisions or refinements over time. 
Academic program faculty used a variety of assessment methods, including commercially available and 
standardized national exams, comprehensive examinations and final projects evaluated by peers, 
faculty, and/or external evaluators, comprehensive research paper or thesis, programmatic growth 
contracts, observations or simulated work experiences, proficiency checklists for writing and speaking, 
and evaluations of student performances and achievement of skills by faculty members and/or external 
evaluators. 

Some faculty believed that the reliability and validity of these assessment tools were increased since 
they were developed by working with numerous professors, external consultants, advisory boards, and 
the Institutional Planning and Research Office. 

A recent initiative (currently embedded within a small number of individual courses) underway at 
Sinclair was a movement labeled “process learning.” Since Sinclair instructors wanted to fully 
embrace the learning college concept, some faculty were adopting learning-centered experiences for 
their students. The underlying philosophy for this approach was that faculty believed they could 
develop stronger student learners who would know how to continuously improve their own learning 
processes. The instructor’s role changed from being the authority figure to facilitating student-centered 
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learning experiences. This was an important educational development because the participating faculty 
were essentially defining specific competencies for individual courses in the following areas: applied 
critical thinking, problem-based learning, cooperative teams, and communications. Faculty used 
different assessment methods to accompany their new teaching approaches. Examples of assessment 
methods included journal writing, case studies, role plays, guided discoveries, and project work. These 
formative assessments may be done by faculty, student peers, or the individual student. Portfolios were 
also being used by some faculty. 



Documenting Student Achievement 

Faculty were required to share their progress and assessment results on a regular basis. This 
information was gathered through a set of interviews that were done with each academic department 
every 2 years. Trained faculty members conducted the interviews and asked faculty to respond to six 
specific areas: 

• Learning outcomes; 

• Curriculum information; 

• Assessment methods; 

• Assessment results; 

• Analysis and actions taken; and 

• General education linkages. 



The learning outcomes were a series of specific competencies that students were expected to master. In 
addition, the general education linkages also focused on liberal learning competencies such as 
communications or critical thinking. Most program faculty used multiple methods to assess these 
outcomes, and they were frequently embedded into required courses. Program faculty articulated then- 
assessment results. The majority of assessment results revealed that faculty were “on target” and that 
no changes or improvements were necessary. Students reported that the writing and speech checklists 
were very helpful, but it was not clear how they may strengthen these particular outcomes. 

Sinclair decided to use interviews rather than surveys as the main vehicle to collect assessment 
information. They found that the results were more useful and the process made the reporting less 
burdensome for the faculty. The interviewers wrote the reports and then asked the individual 
departments to review them for accuracy and to insure the validity of the content. The college was 
getting higher quality information as a result of this interview process used to document student 
learning outcomes. 

An assessment steering committee oversaw the assessment plans and results. This committee worked 
with the leadership of a coordinator for assessment who reports to the vice president for instruction. 

Data Ramifications 

Faculty primarily assessed student outcomes by using locally developed approaches. They worked 
with experts on campus and with consultants. They thought that such a process would make the 
methods more reliable and valid. The assessment checklists for writing and oral communications were 
used by some faculty. However, since faculty of each program were using different assessment 
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methods that changed over time, it was difficult to determine the actual reliability and validity of the 
instruments. 

The COMPASS instrument was used to assess the abilities of entering groups of undergraduates. This 
type of assessment could be examined for its reliability and validity since it was used systematically 
each year. In addition, the American College Testing company could also provide information about 
the usefulness of this placement test, and comparisons could be made across institutions if desired. 
These placement tests could be re-administered a second time (after students complete a series of 
college courses) to better understand if students were gaining in their abilities. Such longitudinal 
assessments were not yet evident systemwide at Sinclair Community College. 

Advantages and Challenges Associated with a Competency-Based Educational System 

A major strength of the Sinclair competency-based educational system and its assessment was that 
faculty felt strong ownership for its development and implementation. Getting faculty responsible for 
the leadership in the university-wide assessment committee really helped professors to have a strong 
commitment to making it succeed. Faculty ownership also led to higher motivation. These changes 
were constructed with the assumption that formative assessment plans work best since they can lead to 
internal improvements that faculty more highly value. 

When faculty initially developed and implemented comprehensive assessment plans, it took a 
considerable amount of resources and time. In the early assessment plan, Sinclair’s leaders tried to do 
everything at one time. As these new ideas were introduced, the leadership realized that they were 
moving too quickly. Therefore, they examined their current environment and identified the top 
priorities and how those could be measured. The assessment of general education outcomes was a 
good example of certain priorities being identified by the leadership, and an incremental plan was 
developed to implement these initiatives over several years rather than in a single year. 

Another related challenge was how to account for the change in faculty time. With these new 
assessment developments, faculty spent considerable time revising their course and program plans. 
Sinclair faculty were successful in receiving substantial merit pay based primarily upon the 
documentation of their assessment activities. In addition, a new performance review system was being 
piloted where faculty could choose to have assessment as a major category to demonstrate their success 
when they are reviewed by their peers. 

Recommendations for Other Colleges and Universities 

Faculty believed that there were several important ingredients to create a competency-based 
educational system with accompanying assessments. They asserted that with these critical elements in 
place, colleagues at other institutions could adopt a similar process to develop these types of initiatives. 

Adequate resources for training and development were critical. Faculty frequently attended seminars to 
leam how to write competency statements and to consider a range of new assessment methods. Faculty 
reported that these types of seminars were very important to help them successfully implement their 
new ideas. In addition, release time from courses for faculty to substantially revise their curriculum 
was extremely important. Support from central administration at the senior levels was important too. 
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Faculty also emphasized the importance of pilot testing new ideas with time and opportunities to make 
revisions. As faculty discovered techniques that work well, they may continue using them. However, 
if certain techniques do not work, then they can make changes based upon the assessment results. 



Conclusion 

The faculty at Sinclair Community College worked consistently for nearly a decade to build and 
implement comprehensive assessment plans. Training and development was essential for faculty to 
learn how to clearly articulate the specific competencies that college students should achieve. All 
major program faculty also identified the specific general education competencies (critical thinking and 
communications) that students should gain through study in the areas of specialization. This was a 
major strength of their plan and created continuity or coherence within the total curriculum. Students 
knew as beginning freshmen that the faculty in their major programs would have clear expectations for 
their learning that built upon the liberal learning skills gained in the general education curriculum. 

Some Sinclair faculty shifted their emphasis from instruction and teaching to a focus on producing 
learning. They were experimenting with new assessment methods, particularly tools that are embedded 
within specific courses. To date, there was no systematic information about the reliability and validity 
of the assessment tools. However, as faculty continue their pilot tests and eventually adopt certain 
assessment methods over time, a clearer picture about the specific impact of the competency-based 
curriculum on student learning and development will most likely emerge. 
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Attachment C-l 



SINCLAIR COMMUNITY COLLEGE 
GENERAL EDUCATION WRITING SKILLS CHECKLIST 1 



ASSIGNMENT PROFILE 


YES 


NO 


N/A 


1 . Understand the type of writing required 








2. Determine the purpose of the writing task 








3. Understand the requirements of the writing task 









CONTENT 


YES 


NO 


N/A 


1 . Clear main idea 








2. Stimulating, insightful main idea 








3. Accurate support 








4. Adequate support 








5. In-depth, authoritative support 








6. Critical, analytical, and/or creative support 









ORGANIZATION 


YES 


NO 


N/A 


1 . Clear opening/introductory section 








2. Engaging, stimulating opening 








3. Consistent, well-planned method of organization 








4. Creative, unique method of organization 








5. Transitional sentences and/or paragraphs 








6. Clear closing/concluding section 








7. Engaging, convincing closing 









STYLE 


YES 


NO 


N/A 


1 . Appropriate vocabulary for audience/reader 








2. Effective word choice, diction 








3. Varied sentence structure 








4. Well-crafted, balanced sentences (parallelism) 








5. Effective transitional words and phrases 









1 Material copied with permission of Karen Wells, Sinclair Community College. 
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SINCLAIR COMMUNITY COLLEGE 
GENERAL EDUCATION WRITING SKILLS CHECKLIST (continued) 



MECHANICS 


YES 


NO 


N/A 


1 . Accurate Spelling 








2. Correct Capitalization 








3. Correct Usage 








A. 


Verb Tense 


A. 


A. 


A. 


B. 


Verb Agreement 


B. 


B. 


B. 


C. 


Pronoun Reference 


C. 


C. 


C. 


D. 


Adjective/Adverb Use 


D. 


D. 


D. 


E. 


Sound-Alikes 


E. 


E. 


E. 


4. Correct Punctuation 








A. 


End Marks 


A. 


A. 


A. 


B. 


Semicolons/Colons 


B. 


B. 


B. 


C. 


Quotation Marks 


C. 


C. 


C. 


D. 


Commas 


D. 


D. 


D. 


E. 


Apostrophes 


E. 


E. 


E. 


F. 


Dash, Hyphen, Parentheses 


F. 


F. 


F. 


G. 


No Run-on Sentences 


G. 


G. 


G. 


H. 


No Unintentional Fragments 


H. 


H. 


H. 


5. Formal 










A. 


Presentation/Neatness 


A. 


B. 


C. 


B. 


Documentation 


A. 


B. 


C. 
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SINCLAIR COMMUNITY COLLEGE 
ORAL COMMUNICATION CHECKLIST 2 



PUBLIC SPEAKING SKILLS 


ABSENT 


POOR 


NEEDS 

IMPROVEMENT 


FAIR 


GOOD 


EXCELLENT 


Preparation 

1 . Determine the purpose of oral discourse 


0 


1 


2 


3 


4 


5 


2. Choose a topic and adapt it according to the purpose 
and the audience 


0 


1 


2 


3 


4 


5 


3. Fulfill the purpose of oral discourse by: 














A. formulating a thesis statement 


0 


1 


2 


3 


4 


5 


B. providing adequate support material and 
documentation 


0 


1 


2 


3 


4 


5 


C. selecting a suitable organizational pattern 
(sequence) 


0 


1 


2 


3 


4 


5 


D. demonstrating careful choice of words 


0 


1 


2 


3 


4 


5 


E. providing effective transitions 


0 


1 


2 


3 


4 


5 


F. providing internal summarization 


0 


1 


2 


3 


4 


5 


4. Establish source credibility 


0 


1 


2 


3 


4 


5 


Presentation 

5. Employ vocal variety in rate, pitch and intensity 


0 


1 


2 


3 


4 


5 


6. Articulate clearly 


0 


1 


2 


3 


4 


5 


7. Employ the level of language appropriate to the 
designated audience 


0 


1 


2 


3 


4 


5 


8. Demonstrate nonverbal behavior that supports the 
verbal message 


0 


1 


2 


3 


4 


5 


9. Use of appropriate speaking aids (visual, auditory) 


0 


1 


2 


3 


4 


5 


SMALL GROUP COMMUNICATION SKILLS 




1 Demonstrate understanding of facilitator role 


0 


1 


2 


3 


4 


5 


2. Facilitate a group meeting 


0 


1 


2 


3 


4 


5 


3. Ask appropriate questions 


0 


1 


2 


3 


4 


5 


4. Encourage balanced participation 


0 


1 


2 


3 


4 


5 


5. Demonstrate effective use of agendas 


0 


1 


2 


3 


4 


5 


6. Demonstrate effective collaboration 


0 


1 


2 


3 


4 


5 


7. Utilize idea generation techniques 


0 


1 


2 


3 


4 


5 


8. Utilize appropriate decision making processes 


0 


1 


2 


3 


4 


5 


9. Manage conflict effectively 


0 


1 


2 


3 


4 


5 


10. Recognize nonverbal factors unique to teams 


0 


1 


2 


3 


4 


5 


1 1 . Recognize stages and outcomes of stages in teams 


0 


1 


2 


3 


4 


5 


12. Demonstrate effective informal and formal team roles 


0 


1 


2 


3 


4 


5 


13. Differentiate between task and relationship behaviors 


0 


1 


2 


3 


4 


5 



2 Material copied with permission of Karen Wells, Sinclair Community College. 
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INTERPERSONAL COMMUNICATION SKILLS 


ABSENT 


POOR 


NEEDS 

IMPROVEMENT 


FAIR 


GOOD 


EXCELLENT 


1 . Phrase questions in order to obtain information 


0 


1 


2 


3 


4 


5 


2. Demonstrate understanding of and ability to 

contribute to a climate which is open, descriptive, 
supportive 


0 


1 


2 


3 


4 


5 


3. Verbalize feelings in appropriate situations for 
disclosure 


0 


1 


2 


3 


4 


5 


4. Describe opposing points of view 


0 


1 


2 


3 


4 


5 


5. Paraphrase information given by others 


0 


1 


2 


3 


4 


5 


6. Effectively manage conflicts with others 


0 


1 


. 2 


3 


4 


5 


7. Strive for consensus 


0 


1 


2 


3 


4 


5 


LISTENING SKILLS 














Attending 

1 . Recall basic ideas and details 


0 


1 


2 


3 


4 


5 


2. Recognize main ideas 


0 


1 


2 


3 


4 


5 


3. Identify supporting details 


0 


1 


2 


3 


4 


5 


4. Recognize explicit relationships among ideas 


0 


1 


2 


3 


4 


5 


5. Listen attentively 


0 


1 


2 


3 


4 


5 


Understanding 

6. Discriminate between statements of fact and 
statements of opinion 


0 


1 


2 


3 


4 


5 


7. Distinguish between emotional and logical arguments 


0 


1 


2 


3 


4 


5 


8. Detect bias and prejudice present in an interaction 


0 


1 


2 


3 


4 


5 


9. Recognize the speaker’s frame of reference 


0 


1 


2 


3 


4 


5 


Evaluating 

10. Synthesize and evaluate by drawing logical 
inferences and conclusions 


0 


1 


2 


3 


4 


5 


1 1 . Recall implications and arguments 


0 


1 


2 


3 


4 


5 


12. Recognize discrepancies between the speaker’s 
verbal and nonverbal messages 


0 


1 


2 


3 


4 
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APPENDIX D 



Hagerstown Community College 



Introduction 

The purpose of this case study was to explore the career transcript system implemented at 
Hagerstown Community College. This new career transcript built upon earlier work focused on 
the creation of CD-ROM modules designed to address specific competencies that college students 
should master. The overarching purpose of this initiative was to help college faculty and 
administrators prepare productive technicians. A consortium of organizations collaborated on 
these initiatives with support from the National Science Foundation (NSF). Although work on 
the career transcript was in beginning stages of development and pilot testing, this particular 
initiative was selected to study since it may significantly influence competency-based initiatives 
at other institutions. 

In the fall 1999 semester, five faculty at Hagerstown Community College were interviewed by 
the Working Group consultant. Each individual interview during the site visit ranged from 30 to 
60 minutes. Participants were full-time faculty who were directly involved and provided 
leadership in the development and implementation of the competency-based educational 
experiences at Hagerstown Community College. In addition, the co-principal investigator of this 
work was also interviewed. Numerous documents, including the college catalogue, published 
chapters, and evaluation reports, were reviewed by the Working Group consultant. 



The Setting 

The workplace is changing dramatically. Employees are less supervised and more often expected 
to make decisions that were formerly made by their managers (Camevale, Gainer, and Meltzer, 
1990). Employers want a new type of worker with a broad set of foundation skills at least in the 
basics that will enhance learning on the job. Today manufacturers seek well-trained employees 
who can work effectively in teams to solve real-world problems. In 1995, a consortium was 
formed including the Johns Hopkins University, related industries, a multimedia production 
company, Hagerstown Community College, Modesto Junior College, New Hampshire 
Community and Technical College System, Northern Essex Community College, and South 
Seattle Community College. 

The main purposes of this consortium’s work (funded for 3 years by the National Science 
Foundation as the first phase of work) was to determine how to: 

• Prepare community college graduates for the 21 st century; 

• Help students learn competencies that will remain valuable as technology changes 
and they change jobs and employers; 

• Provide students with a degree that is “portable,” one that is recognized by employers 
in different states; 

• Encourage faculty from many disciplines in different institutions to use electronic 
communicate to collaborate on course construction; and 

• Use CD-ROMs to integrate workplace competencies into general education 
coursework (Packer and Mathias, 1995, p. 38). 
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These purposes guided the development and design of the entire initiative. 



Identifying Important Outcomes 

One of the early issues in this project was reaching an agreement on the specific educational 
outcomes that college graduates should master. For about 1 year, a panel of 17 employers met 
four times with another panel of 15 educators representing the nation’s community colleges 
(Packer and Mathias, 1995). This length of time was necessary in order to promote a full 
discussion and review of potential important outcomes. In addition, the appropriate and 
necessary stakeholder groups were directly participating in the identification and consensus- 
reaching process. The two panels drew upon a number of sources to define standards including 
research from the Secretary’s Commission on Achieving Necessary Skills (SCANS), an existing 
DACUM (Developing a Curriculum in Manufacturing) report, and the skills-setting efforts of the 
participating industry representatives (Packer and Mathias, 1995). 

These panels suggested two sets of recommendations. First, they recommended that any 
curriculum should be: 

• Able to serve diverse populations, including underemployed older workers; 

• Consistent with curricula of both high schools and colleges; 

• Integrated with existing academic programs in the current community college 
system; and 

• Able to use technology in delivering instruction (Packer and Mathias, 1995, p. 40). 

The second recommendation was that 22 workplace competencies should be taught within 16 
different modules. The SCANS competencies included three foundation skills and personal 
qualities, as well as five workplace skills that are necessary for effective performance on the job: 

• Foundation Skills 

Basic skills: Reading, writing, arithmetic and mathematics, and speaking and 
listening. 

Thinking skills: To reason, to think creatively, to make decisions, and to solve 
problems. 

Personal qualities: Individual responsibility and self-management, and sociability 
and integrity. 



• Workplace Competencies 

Resources: How to allocate time, money, materials, space, and staff; 

Interpersonal skill: Work on teams, teach others, serve customers, negotiate, and 
work well with people from culturally diverse backgrounds. 

Information: Acquire and evaluate data, organize and maintain files, interpret and 
communicate, and use computers to process information. 
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Systems: Understand social, organizational, and technological systems, monitor 
and correct performance and design or improve systems. 

Technology: Select equipment and tools, apply technology to specific tasks, and 
maintain and troubleshoot equipment (SCANS, 1992). 

Ultimately, this NSF project focused on covering SCANS competencies through five CD-ROM 
modules assigned to a particular academic home. 



Creating CD-ROM Modules to Address Important Competencies 

Five lead college teams (from the participating institutions in the consortium whose disciplines 
include mathematics, information science, English and technical communication, science, 
business, and technology) had responsibility for developing, designing, and implementing 
modules with guidance from John Hopkins University. The multidisciplinary nature of each 
working group strengthened cross-disciplinary learning within each module (Packer and Mathias, 
1995). Some teams substantially designed and programmed modules, while other teams 
primarily served in an advisory review capacity for ideas generated and designed by John 
Hopkins University. Most faculty helped design the modules to include course content. 

The five CD-ROM modules included (1) Designing the Electric Car: Selecting from Alternatives 
to Maximize Profit; (2) Improving the Quality: Putting Together a Problem Solving Team; (3) 
Making Complex Decisions: Using Computer Modeling and Decision Matrices; (4) Using 
Statistical Quality Control: Monitoring and Correcting Performance to “Build Quality In”; and (5) 
Using a Problem Solving Process to Improve and Design a System: Making an Engineering 
Change Orders Paperless. 

Each module was embedded into the appropriate existing course. One module was designed for 
mathematics, one for science, two for technology (engineering and computer), and one for a 
communication course. The CD-ROMs and accompanying materials for each module constituted 
about 9 hours of a particular course or 20 percent of a three-credit semester course. Each module 
was designed by faculty to teach some of the course content in math, science, or communications. 
Also, each particular module was created to teach certain skills outlined in the SCANS 
competencies or standards associated with the module. 

For each particular module, faculty teams identified one or two primary SCANS skills that were 
supposed to be addressed. In some cases involving complex problems, additional SCANS skills 
could be learned by students. Faculty designed real-life scenarios or cases that students solve. 
These learning experiences focused on authentic problems that frequently occur in industry. The 
CD-ROM technology offered scenarios that present open-ended problems where students must 
typically draw upon knowledge from several disciplines. Instructors agreed upon a particular 
problem with enough academic content to support the student research and to justify placing the 
module within a particular discipline and course (Mathias, 1999). Students typically worked in 
teams to research, organize and present data, write reports, and give presentations justifying why 
they made certain decisions. The CD-ROMs also contained tutorials for students who wanted to 
review information about certain pertinent subjects such as statistics or team roles. 
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Assessment Strategies for Course-Embedded Modules 

The SCANS outcomes included process skills that can be more difficult for faculty to assess 
within the traditional classroom. Instructors frequently construct multiple-choice tests to measure 
recall or understanding of material. However, certain process skills such as working effectively 
in a team or making a strong oral presentation require faculty to observe performance and thus 
make the assessment more authentic. In addition, the cases within each module require students 
to apply knowledge and solve complex problems. 

Faculty worked with AES, International (a commercial company) and the SCANS competencies 
to develop behavioral indicators that documented performance mastery of certain outcomes. The 
instructors prepared lists of tasks and the actual behaviors they expected of students completing 
the projects as outlined in the CD-ROM learning experiences. Then the commercial company 
translated these behaviors into an assessment scoring rubric. Faculty were taught about how 
rubrics were constructed and then used as a means of documenting student performance. Faculty 
with experience over time may then refine these behavioral indicators. In effect, these indicators 
defined successful student performance. Typically, three of these types of assessments would be 
completed for students within each particular module. 

Attachment D-l includes an example of an actual student performance assessment method. As 
students solved an advertising problem (in this particular example), they were expected to 
demonstrate their command of the subject that composes the top categories on this assessment 
form. Faculty, students themselves, or their peers can then evaluate their performance by rating 
the strength of their knowledge with a range of no understanding to expert understanding. The 
problem-solving skills were defined by the competencies at the bottom of the form. These 
competencies were identical for all modules addressing problem solving. Typically, there were 
five to seven competency statements listed. Other modules may address working in teams, 
interpreting and communicating information, and making a presentation. Again, evaluators rated 
the degree of mastery. In the future, the criteria listed at the bottom of the form may become a 
mixture of different SCANS outcomes rather than only one particular outcome, such as problem 
solving. 

Some outcomes are interrelated rather than discrete units. Although faculty teams worked to 
isolate tasks and align them with certain individual outcomes, it is possible that students may 
demonstrate their ability to work in teams as well as their ability to work with diversity 
simultaneously. On one specific task, faculty could then evaluate student performance in 
working with teams and working with diversity. Faculty teams plan to examine other behavioral 
indicators for SCANS outcomes and test them by linking outcomes to specific tasks within the 
learning modules. 

A second form of assessment consisting of videos was developed by another commercial 
company. These assessments were designed to evaluate certain SCANS competencies. Students 
were presented with a scenario in the workplace that simulates a real problem. Then, through a 
multiple-choice assessment, students selected the most appropriate answer that was evaluated by 
the commercial company. Some faculty found these video assessments to be less useful since 
they did not develop them. Also, some faculty did not believe that the videos related directly to 
their own objectives for their particular course. 
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Documenting Student Achievement on the Career Transcript 

As this new system incorporating project-based modules was embedded into the curriculum, 
faculty and employers strongly believed that the traditional transcript with letter grades was not 
sufficient to demonstrate students’ mastery of key concepts and SCANS competencies. 
Therefore, a career transcript was designed by the Johns Hopkins University team and a 
commercial company to indicate students’ specific levels of achievement. The commercial 
company had substantial experience in developing software to help organizations’ human 
resource departments make hiring decisions. Through the second cycle of 3-year funding from 
NSF, project leaders developed and began testing an Internet-based career transcript in 1999. 
That career transcript was designed to supplement the more traditional college transcript that 
usually indicates course completions and letter grades. 

In individual courses, faculty assessed students as they completed their project-based learning 
tasks within the specified modules. The main assessment results were initially noted on the 
special assessment form (see attachment D-l), and then these results were placed into the Internet 
database career transcript. Students could review and print out their transcripts and decide what 
information they wanted to share with employers. In addition, students decided what assessment 
results should go on the career transcript. If students perform well, they will most likely want the 
results posted on the transcript. However, if students do not perform well, they can decide that 
certain information should not be included on the transcript. Students may also decide to block 
certain portions of the transcript from employers in order to tailor the information to the 
requirements of the particular position that they are seeking. Also, when students repeat certain 
skills and their assessments in other modules, then only the best efforts will be shown on the 
career transcript. In the future, it is expected that supervisors at the workplace could also assess 
performance and then indicate those results on the career transcript. Attachment D-2 includes an 
example of a career transcript that is organized like a resume by listing the SCANS competencies 
and levels of achievement. 

This career transcript is premised upon the hypothetical resume outlined in a previous SCANS 
report (1992). The current template for the career transcript was modified as it emerged through 
discussions between the Johns Hopkins University team and the commercial partner (Alignmark). 
Future designs of the career transcript will be revised based upon faculty feedback. 



Data Ramifications 

The career transcript system and its links to the new assessments of student performance in CD- 
ROM modules were at the beginning stages of development and testing when this case study was 
conducted. Therefore, the full implications of the data ramifications were not evident. Project 
teams plan to collect more data over time (as this initiative expands and is refined) that may 
provide some evidence about the reliability and validity of the assessment methods. 

When sufficient numbers of students participate in these new initiatives, the project teams will 
begin to make comparisons between a control group (which does not participate in these 
initiatives) and those who do fully participate. Such comparisons might include an examination 
into whether participating students have higher graduation rates than those who do not participate. 
Another comparison between these two groups of students may examine course completion rates 
and their particular levels of competence. Once graduates are in the workplace or in internships, 
employers may also be asked to evaluate students’ performance on the job. 
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Advantages and Challenges Associated with a Competency-Based CD-ROMs and Career 
Transcripts 

Several advantages were associated with the development and creation of CD-ROM learning 
experiences that are linked with career transcripts. From evaluation reports, it was clear that 
students believed they were achieving stronger learning outcomes. Approximately half of the 
students indicated that they learned new communication and teamwork skills, while two-thirds 
stated that working through the CD-ROM exercise helped them to strengthen their presentation 
skills (Gold, 1998). About two-fifths of the students believed they made gains in their critical 
thinking abilities as well as evaluating different courses of action within a larger context and 
learning to gather information and analyze it for decision making (Gold, 1998). About four- fifths 
of the students reported that the module taught concepts or skills that they would have not learned 
in traditional classes in which they enroll. In particular, students stated that they rarely worked in 
teams in their traditional college classes (that were not part of this project), even though they did 
gain teamwork experience in the workplace (Gold, 1998). Since the career transcript was very 
new, in 1999, there were no systematic results yet about its usefulness from the students’ 
perspectives. 

Faculty believed that the CD-ROM learning experiences fostered closer connections between the 
worlds of work and academics. Through the case-study approach, students worked in teams to 
solve real-world issues. These learning experiences also encouraged students to use technology 
more in the classroom, as well as to use their own college computer facilities. 

Faculty initially had some concerns and reservations about the career transcript. There was 
typically only one CD-ROM project-based learning module in a particular course. At the time the 
case study was conducted, a very small number of faculty used these learning modules at 
individual institutions. Frequently, only student outcomes from this particular module were 
assessed for subject matter and SCANS competencies that were then placed on the career 
transcript. If this module were the last learning experience within a course, it would serve as a 
“capstone” experience and faculty would feel comfortable in relying on the assessment results in 
the transcript. However, the modules can be embedded at any point from the beginning to the 
middle of the course, too. Therefore, some faculty believed that the career transcript only shows 
student performance for one-fifth of the course. Some faculty were concerned that employers 
may think that the results are based upon extensive assessments across cumulative learning 
experiences. This challenge was illustrated at Hagerstown Community College, where only two 
students had taken more than one module for which results could be indicated on the career 
transcript. In addition according to some faculty, employers were confused about how students 
were assessed, which potentially leads to difficulties in making interpretations about the 
assessment results. 

Some faculty did not clearly understand the purposes of the career transcript. Perhaps since this 
was such a new part of the initiative, some faculty did not fully comprehend how the career 
transcript would be developed and what its potential usefulness was. However, some stated they 
would like to have more input into the actual development of this transcript. 

The format of the career transcript was still emerging. It appeared that the leaders of this project 
were working to get feedback from both faculty and employers about the structure, organization, 
and content of the career transcript to make it useful and meaningful. 

Another challenge for faculty at Hagerstown Community College was that this type of innovative 
work had not been widely supported by their colleagues at their own institution. Some faculty 
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did not see a need for substantial change since they feel fully committed with heavy teaching 
loads and major student advising responsibilities. Reasons cited for this lack of widespread 
change to a more project-based learning approach by some of the instructors included the belief 
that the lecture-centered approach works best and the feeling that it is very difficult to incorporate 
these types of modules into a traditional classroom. Faculty were sometimes reluctant to 
integrate the CD-ROMs into courses that were already fully programmed, especially when these 
types of projects were only viewed to tangentially address the academic material presented by 
some faculty (Gold, 1998). 

Related concerns exist about the major amount of time to revise a course into a student-centered 
approach that uses cases or project-based learning experiences. Potential faculty adopters may 
perceive this type of initiative to be a substantial cost that outweighs the benefit. Although only a 
small nucleus of faculty from Hagerstown Community College were involved in this project, they 
did report ongoing discussions with their peers in an effort to convince them of its usefulness and 
value. Senior administrators were convinced of its importance and provided release time from 
courses for faculty to design and implement the modules into existing courses and to evaluate 
them. 

Another challenge for this particular effort is that faculty needed updated and state-of-the-art 
computers. Although students usually have access to current technology, some faculty (from the 
most recent faculty development seminars) stated that they have old and outdated technology. 
Some faculty also lacked adequate support from their technology experts on campus. This 
created a potential barrier for faculty trying to adopt these innovative CD-ROM learning 
experiences. Fortunately, faculty at Hagerstown Community College had appropriate computer 
hardware and technology available. However, the technology issues were challenging for faculty 
at other institutions who were trying to adopt the CD-ROM learning experiences. 



Recommendations for Other Colleges and Universities 

A strong faculty development plan was cited by professors as extremely important to the success 
of building competency-based modules. The consortium under the leadership of Johns Hopkins 
University designed a sophisticated professional development program for individuals interested 
in implementing these transferable modules into their own courses. The main purpose of this 
program was to disseminate these modules to professors at similar colleges and to prepare faculty 
teams to become change agents at their own institution. Teams usually included four or more 
individuals and one senior-level administrator. 

These change agents will decide which particular existing modules (developed through the NSF 
grant) can be incorporated into their own courses and then train others in “new ways of 
instructing, assessing, documenting, and reporting student learning and competency achievement” 
(Mathias, 1999a, p.l). The vision was that these new institutional partners will actually 
implement the CD-ROM modules into certain courses including the use of standard assessment 
instruments and build career transcripts for participating students. All institutional partners 
would continue implementation and documentation of student learning outcomes through the end 
of the project in August 2001 . 

The faculty development program currently extended over an 8-month period. Participants 
worked through guided independent learning activities. They received overviews of the CD- 
ROM case studies so that they could make informed decisions about which module was most 
appropriate for their particular courses. Faculty reviewed learning modules to better understand 
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the content, organization, and technology associated with its delivery. They were also mentored 
throughout this development stage through a Web-Based Electronic Learning Community, which 
includes face-to-face video conferencing sessions (Mathias, 1999b). This approach was designed 
to help faculty successfully navigate a challenging change process. Electronic mentoring offered 
participants time to share insights, challenges, course outlines, and best practices. 

Participants were also expected to attend a 3-day seminar offered by members of the consortium 
who worked with 16 different college teams from across the United States. Substantial time was 
devoted to helping faculty understand their changing roles from the authoritative expert to the 
project-based and team-oriented facilitator who help students learn to collaborate. In addition, 
faculty usually needed assistance identifying important competencies and designing the 
appropriate assessment methods. In summary, the main topics at this seminar included how to 
blend complex, industry-based problems into traditional academic coursework, manage students 
working in teams, facilitate problem-based learning, integrate technology, and assess and 
document student achievement using the SCANS competencies (Mathias, 1999). 

Strong support from senior central administrators was critical. Faculty strongly believed that 
release time from courses was essential as instructors strived to dramatically change then- 
traditional classrooms. Without release time, faculty did not get sufficient time to develop and 
revise their courses. These changes were very substantial for faculty, and they needed the time to 
reflect, develop new ideas, and get feedback on their plans from their colleagues. 



Conclusion 

The faculty who were engaged in this innovative work were highly motivated to change their 
classrooms into a more project-based and team-oriented environment. Through major training 
initiatives, faculty were encouraged to think about new ways to work with students and develop 
real-world problems with multiple solutions rather than the one “right answer.” Faculty 
successfully developed new learning experiences using CD-ROMs. They were disseminating this 
information to other colleagues and training new adopters. The modules were transportable into 
the appropriate courses at other colleges and universities. Transportability was also enhanced 
since modules built upon the SCANS (1992) competencies that have widespread acceptance in 
certain professional fields. 

The creation of the career transcript has the potential to build stronger bridges among faculty, 
students, employers, and supervisors of internship or clinical experiences. The articulation of 
specific levels of achievement mastered by students will be important to share with multiple 
audiences. Employers will have more useful information to make judgements about the quality 
of their potential employees. This endeavor was very new and will take some time to gain 
acceptance about its value and use. However, ongoing consultations with all affected 
constituencies should increase the likelihood of its success. 
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Attachment D-l 




AES SkiU@ 


HAGERSTOWN COMMUNITY COLLEGE 
STUDENT PERFORMANCE ASSESSMENT 1 


COACH 


Date: 


Assessed Bv: Sunervisor Self 


Peer 


Other 


Student Name: 




Student ID # 



Campus Department Program Sub-Set Faculty. 

Carson: Course: Mathematics Objective: Entrepreneurial Mathematics CD 



Outcome: Solve Advertising Problem 



Outcome Scoring Key: Command of Subject 

l=No Understanding 2=Little Understanding 3=Basic Understanding 4=Sufficient Understanding 
5=Expert 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Determines objective statement 

Defines constraints 

States problem mathematically 

Employs CAS and/or spreadsheets for calculating successful solution 
Applies the above 4 steps for each different media type 
Converts data to sales projections 
Determines alternative solutions for other price points 



Process Scoring Key: Command of Subject 

l=No Understanding 2=Little Understanding 3=Basic Understanding 4=Sufficient 
Understanding 5=Expert 

Level of Difficulty: 1 2 3 4 5 
Process Skill: F9 - Problem Solving 

INSTRUCTIONS: Fill in the number which best describes the degree of effort demonstrated by 
the student in this task. 



1 2 3 4 5 


Clearly defines the problem 


1 2 3 4 5 


Seeks probable causes of the problem 


1 2 3 4 5 


Sets criteria for successful solution of the problem 


1 2 3 4 5 


Considers alternative solutions 


1 2 3 4 5 


Identifies likely outcomes for each alternative 


1 2 3 4 5 


Implements an alternative that meets selected criteria 


1 2 3 4 5 


Monitors progress of proposed solution 



1 Material copied with permission of Elizabeth Mathias. 
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Attachment D-2 



Hagerstown Community College 
SCANS Transcript for 

Sample Tester 

As of 7/14/99 2 



SCANS Skills 



1 Resources 


Interpersonal Skills 


Information 


Systems | 


| Technology 


Thinking Skills 


Personal Qualities 


1 



SCANS 

Skill 


Date 

Test 

Taken 


Source of Information 


Performance 










Allocates Time 




7/14/99 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, prepares and 
organizes multiple schedules, manages timelines, 
and recommends timeline adjustments, with 80-89 
percent proficiency. 


Allocates Materials 




7/14/99 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, orders and 
maintains inventory, and monitors safe and 
efficient utilization of materials, with 70-79 
percent proficiency. 


Allocates Space 




6/25/99 


Nurses Aide, Johns 
Hopkins Hospital 


Noticed that the office arrangement impeded 
patient traffic flow in doctor’s office and 
rearranged the furniture to accommodate a better 
flow. 


Teamwork 




1/23/98 


AccuVision WSS 


For Actively acknowledges and recognizes the 
positive work efforts and accomplishments of 
team members this participant performs 
acceptably. 


General Interpersonal Skills 




1/23/98 


AccuVision WSS 


The performance of this individual on the 
AccuVision WSS assessment indicates a 90 
percent probability of success in Interacting with 
Others. 



2 Material copied with permission of Elizabeth Mathias. 
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Hagerstown Community College 
SCANS Transcript (continued) 



Acquire/Evaluate Data 




7/14/99 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, analyzes data, 
Integrates multiple items of data, and contrasts 
conflicting data, with 80-89 percent proficiency 


Identify, Understand, and 


Work w/Systems 




7/14/99 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, analyzes system 
configuration/stability, and recognizes system 
strengths/limitations, with 70-79 percent 
proficiency. 


Maintain & Troubleshoot Equipment 




7/14/99 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, implements 
technological improvements/changes and 
generates technological solutions, with 60-69 
percent proficiency. 


Reasoning 




1/23/98 


Nurses Aide, Johns 
Hopkins Hospital 


Doctor noted client’s creative thinking skills when 
addressing problems and developing solutions. 


Responsibility 




1/23/98 


AES International 
Skill Coach 
Assessment 


For an Entry Level Employee, monitors 
performance standards and follows up on assigned 
tasks, with 90- 1 00 percent proficiency. 
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APPENDIX E 



Community Colleges of Colorado Incumbent Worker Project 



Introduction 

The purpose of this case study is to explore the beginning development of the Incumbent Worker 
project funded by the United States Department of Labor as one of 12 national demonstration 
grants. A consortium of community colleges and businesses are formally collaborating to address 
the employment needs within the state of Colorado. This site was selected since the training 
initiatives will include competencies and performance objectives for employees. 

In spring 2000, four individuals were interviewed about this particular initiative. The participants 
included individuals who directly provide leadership for the planning and implementation phases 
of work associated with the Incumbent Worker Project. Each individual interview during the site 
visit ranged from 30 to 60 minutes. The Working Group consultant reviewed numerous 
documents including the original grant proposal and information outlined in private Web pages. 
Since much of the work is still under development and planning, there were limited materials to 
review at the time of the case study. 



The Setting 

Employers want a new kind of professional with a broad set of workplace skills and a strong 
foundation in the basics that will facilitate learning and retention in new positions. Deficiencies 
in many basic skills (including effective writing, speaking, or reading) are barriers for entry-level 
employees and dislocated workers attempting to adapt to economic and technological change 
within diverse companies (Camevale, Gainer, and Melzer, 1990). Employers believe there is 
increasingly a gap between the skills that they need from their new employees and the actual 
skills that such employees demonstrate. In reality, many new employees and dislocated workers 
do not meet the requirements of their new jobs and particularly lack the basic skills. 

Employees are challenged to improve their basic skills through training opportunities to become 
competent in their jobs. Some major reports indicate that the expectations for effective 
performance are much higher today than in previous decades due to major changes in business 
environments (Business-Higher Education Forum, 1997; Camevale, Gainer, and Meltzer, 1990). 
Therefore, the gap between the skills required for effective performance and the actual 
performance of employees is increasingly wider (Business-Higher Education Forum, 1997). 

Colorado’s economy reflects the issues that face many states today. In fall 1998, employers who 
participated in the Denver Workforce Summit reported that their main concern was the 
weaknesses in employees’ basic skills and the shortage of qualified workers for job openings 
(Community Colleges of Colorado, 1999). They found that 7,000 technology jobs are unfilled in 
the state, and that number is projected to reach 30,000 in the next decade. They also reported 
major shortages in the supply of trained workers in the fields of technology, advanced 
manufacturing, and telecommunications as well as other technical jobs. 

The Community Colleges of Colorado (CCC) is providing leadership for this 2-year, Department 
of Labor grant. CCC consists of 14 partner colleges with the local governing entities, under a 
central office with an Executive Director and Staff. Each of these colleges has a training division 
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and an instructional division that designs and delivers learning to corporate and small business 
customers. 

The grant has two major purposes. First, training services will be offered directly to incumbent 
workers at certain companies. Second, a state-level college training support system will enhance 
local company/employees services. Leaders of the grant expect that at least 4,350 incumbent 
workers will receive necessary services. Each community college is in a strong position to work 
collaboratively with industries since they each have a business service unit on campus that 
includes experienced assessment and training staff and managers (CCC, 1999). 

The targeted population of potential participants for training opportunities (designed by the 
community colleges in Colorado) include the following: 

• New and recent hires; 

• Employees who have been targeted for retention attention; 

• Non-English-speaking employees; 

• Employees who will lose employment when the company upgrades to electronic 
systems; 

• Employees with obsolete skills; 

• Employees who can be promoted if they learn new technical skills required by the 
company; 

« Employees who lack basic reading, writing, math skills and therefore cannot be 
trained; and 

• Employees whose hourly wage is less than the company average per grade. 



CCC has named 9 of its system’s community colleges and 18 different companies to address 
employment needs in Colorado. Each of the nine community colleges has established a training 
relationship with one or more local companies to offer specific types of classes or modules for 
specific employers. For example, the Community College of Denver is creating modules on 
business writing, career development, resume writing, interviewing techniques, communications 
with a diverse workplace, customer service, telephone skills, time management, team building, 
and conversational Spanish for Children’s Hospital. The Community College of Aurora is 
focused on customer service for AT&T Cable Services. Pueblo Community College is 
responsible for designing basic computer applications learning experiences for Parkview Medical 
Center. Front Range Community College is working with Sun Micro Systems on the Star Office 
(a particular type of computer application suite that includes word processing). 

Specific criteria for making decisions about participating companies for this particular project 
included the following: 

• Show evidence of commitment to employee development, particularly if they had a 
series of programs already initiated; 

• Provide matching funds and other resources to the project; 

• Work collaboratively with the community college and with other companies; and 
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• Experience incumbent worker development problems that could be reasonably 
addressed by the project (for example, high turnover rates). 

In 75 percent of these companies, more than half of the employees are targeted for assistance, 
suggesting a sizable commitment. Interviews with company personnel indicate that as many as 
45 percent of these employees are severely lacking the basic skills and/or English proficiency 
(CCC, 1999). 

A major portion of the grant is dedicated to creating the Colorado State Training Solutions Center 
(TSC). Business training professionals from the community colleges have found that training and 
services must be delivered more quickly and in more varied delivery formats to better meet the 
growing training needs of local companies and the many different learning styles of employees 
(CCC, 1999). Typically, most professional staff at colleges have very little time to create or 
modify training modules since they are primarily responsible for managing training sessions and 
implementing assessments. The TSC is to address employment issues by overseeing the creation, 
revision, and delivery of training modules as soon as they are needed by different companies. 

TSC proposes an electronic storage and retrieval system so that partners can have access to a 
wider range of materials for their use and modification. Since these materials are modularized 
and their delivery formats will be standardized, partners will be able to draw upon and assemble 
different types of modules into new course offerings. 

This project will provide for the creation of the standardized formats for the training modules and 
distribution process. Instructional design staff (often from the community colleges) will work 
with the specific training division and the company, as is typically the case for customized 
training. Designers from partner colleges will create and test modules during the grant period. 
The content that each partner will contribute is the result of training needs determined with the 
partner companies named in the grant. This center could help college training divisions respond 
more quickly and efficiently to the needs of employers in the state. 

TSC is housed within the CCC Career and Technical Education Resource Library located at the 
Higher Education Advanced Technology Center on the Lowry Air Force Base Campus in Denver. 
By offering customized training materials, advanced delivery system solutions, and train-the- 
trainer opportunities, it will help local college workforce-training divisions and their workforce 
partners to be more effective in working with incumbent workers and their employers (CCC, 
1999). 



Identifying Important Outcomes 

The Training Solutions Center Curriculum System is envisioned to consist of a competency- 
based, modularized curriculum and an electronic storage and retrieval system of workplace 
learning and performance materials (CCC, 1999). The electronic and retrieval system is 
anticipated to support both group and independent-learner models with various formats including 
instructor-led classroom training, Web-based instruction and performance support, CD-ROM, 
video, and audio. 

The leaders describe their vision for the competency-based curriculum as a model that defines the 
creation and delivery of learning content through stated expectations of learning and performance 
and provides for measurement of the results. Competency is specifically defined as the 
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“knowledge, skill, or attitude that enables one to effectively perform the activities of a given 
occupation, or function to the standards expected in employment” (The International Board of 
Standards for Training and Performance and Instruction: http://www.ibstpi.org/idrevised.html). 
A performance objective is defined as what the learner can expect to do as the result of a training 
module (show knowledge or demonstrate a task). It is typically a more detailed level of learning 
and performance than a competency. It defines what performance can be expected. It also names 
conditions of performance and is stated in terms that are measurable and observable (see 
attachment E-l). Modules are units of learning defined by the performance objectives. Courses 
are a set of modules presented as a unified learning experience. 

The training divisions at the community colleges will identify the important outcomes needed for 
the training experiences, often through multiple methods. Staff will make phone calls to business 
managers or conduct interviews and focus groups as well as survey the employees at the specific 
companies about their training needs. Interviewed participants for this case study primarily 
identified training needs with business managers or training units through a few phone 
conversations. 

At the time of the interviews, project staff were constructing a sample of modules. Most of these 
learning experiences will be refined when they are tested in summer 2000. The vision is that 
most of these modules will vary from taking about 15 minutes to 1 hour to complete. It is 
anticipated that employees could take several modules to address specific needs. Some training 
experiences could be 1 day in length, typically running from 4 to 6 hours. For most modules, 
employees would either pass or fail. While single modules will not have academic credit 
attached, it is possible that a series of related modules may lead to either a certificate of 
successful completion or the award of academic credit. 

Currently this project is moving toward developing stated competencies. Each module begins 
with the identification of performance objectives. At a future date, performance objectives will 
be linked and articulated back to academic competencies to facilitate the assignment of potential 
academic credit. As businesses become more experienced in developing company learning 
around position-based competencies, performance objectives can be expressed in relation to 
company or industry-defined competencies. 

Leaders of the Incumbent Worker Project are creating a curriculum design process that involves 
eight major steps leading to the implementation of training modules (see 
http://www.coloradotraining.org). The TSC will maintain and oversee the development of these 
modules. The majority of modules are currently under development. 

The major steps in this process are to: 

• Conduct a training needs assessment with the customer (frequently through 
interviews with the client); 

• Apply the learning domain (cognitive, affective, and/or psychomotor) concepts to 
training needs; 

• Apply task analysis to desired outcomes and determine prerequisite modules; 

• Write a performance objective for the module; 

• Identify assessment (measurement) of performance objective; 
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• Determine content that will allow the learner to successfully achieve the performance 
objectiVe; 

• Apply process for designing module layout and design; and 

• Review and revise materials. 



Ideally, each instructor-led module must explain or address the following points: 

• Identification of the audience it is designed for (manager, technical staff, customer 
service, or front-line/entry-level employees); 

• Curriculum format (print/instructor-led in classroom, PowerPoint, Web, CD-ROM, 
video, audio, book, part of course, module only, or other format); 

• Title of module; 

• Developer’s notes (required to indicate a description of the module including the 
performance objectives being drawn upon); 

• Prerequisites (including specific baseline skills in addition to prerequisite 
modules); 

• Complete module outline (including content with list of terms and their definitions); 

• Assessment methods; and 

• Scoring guide (provides correct answers for objective tests or observable 
characteristics necessary for satisfactory performance in achievement tasks). 



Assessment Strategies for Modules 

Assessments are designed by the training curriculum designers and are intended to measure 
student learning before and after the delivery of the module. Types of learning assessments will 
include paper-pencil tests (e.g., multiple choice, true/false, and short answer) or demonstrations 
of learning through applied tasks (see attachment E-l for an example currently under 
development). 

During the training experience, participants will be asked for feedback about whether the training 
material matches their needs. If it does not match, then the trainer will be able to revise the 
learning experience to better fit employees’ needs. At the end of the training, formal evaluations 
will be conducted to determine the value of the learning and employees’ satisfaction. Some 
partners may also do followup evaluations at the workplace 1 month after the training by asking 
employees about changes in their behaviors (related to the purposes of the training) and by asking 
supervisors about changes in the employees’ performance. 

Project staff expect formative assessments to provide learners with results and feedback that can 
be used to determine progress with material. Summative assessments of employees will indicate 
passing results or failing scores. Other potential measures to evaluate the impact of the grant 
itself include employees’ retention rates in their positions (for those who participated in the 
training experiences) and employees’ rates of promotion or wage gains within the company. 
Validity and reliability are issues that the curriculum leaders are beginning to consider, and 
relevant data will be studied as the assessments are implemented later this year. 
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Advantages and Challenges Associated with Systemwide Shared Training Modules 



The project leaders emphasized that research demonstrates that performance-based learning is 
very important to employers for the results it produces in the workplace. The modularized 
approach that has been created should help the trainers to develop stronger learning experiences 
that respond more quickly and efficiently to the changing needs of the workplace. Separating the 
learning experiences into modules allows for timely revisions of content as needed and enhances 
the tailoring or customizing of training for particular companies. As project leaders fully 
implement this initiative in the summer 2000, they expect that individual modules will be 
transportable to meet the needs of different companies or other community colleges. Since this 
project is at the beginning stages of planning, there are no data or assessment results yet to 
determine the particular dimensions that are working well and what could be improved. 

Several challenges are associated with the planning for this ambitious initiative. Individuals who 
are developing modules have seldom systematically thought about all of the required segments as 
outlined in the curriculum planning process. Prior to this project, none of the individuals wrote 
specific performance objectives or directly linked those objectives with the design of 
assessments. The project leaders did provide classes for designers to learn about the design 
concepts related to the curriculum. After designers submitted an initial module, they participated 
in an individual feedback session with project leaders. 

These are new learning experiences for the curriculum developers, who reported some challenges 
in generating the required parts for each module. The, leaders of this project identified the 
competency-based model with performance assessments as the key to delivering the types of 
learning experiences that employees need to become improved workers. This model has posed 
some challenges, particularly at the implementation phase, because the training divisions in the 
community colleges are trying to address the individual business performance employment 
system needs, industry standards, and vocational skill standards for higher education programs. 
In addition, CCC is establishing common course numberings across the system’s 14 colleges that 
may result in the establishment of common course competencies. 

Some individuals are concerned that there will not be enough iterations to test the modules before 
they are fully implemented. In a private Web page designed only for project participants (at this 
time), numerous Web resources are outlined to help instruct individuals about how to document 
their modules. In addition, specific terms are defined and examples are provided for each 
required reporting element (such as performance objectives and assessment). The project leaders 
provide curriculum planners with feedback about how to strengthen their modules. 

Curriculum planners want to develop performance objectives that both employers and their 
employees will understand. However, another challenge is the articulation of performance 
objectives that will be transferable and not solely dependent upon a particular module useful in 
only one corporate context. The goal for the curriculum planners is that these experiences will be 
transferable, but employers tend to mainly consider their own context. In addition, curriculum 
planners want the performance objectives to be specific enough to guide the learning experiences 
as well as the appropriate assessments. At times, employers may use broader language, but the 
curriculum planners must take broad concepts and translate them into useful performance 
objectives. 

Individuals also reported that this curriculum planning process takes considerable time. There is 
some concern that not enough resources have been allocated to support the development of these 
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modules. Some individuals report doing this work (including the administration and 
coordination) as an overload and are concerned that this may affect the quality of their outcomes. 

Community college presidents signed formal letters agreeing that the curriculum developed 
through this initiative would be owned by the project. Although participating colleges agreed to 
share their curricula, there are still issues that need to be resolved when employers develop their 
own proprietary curricula. The Training Solutions Center will accept generalized content that is 
not specifically linked to individual businesses. But in order to be part of this initiative, 
individuals must agree to share their modules. 



Recommendations for Other Colleges and Universities 

An important recommendation from project leaders is the creation of an environment where risk 
taking is encouraged and supported. Such an environment needs to foster collaborations by 
working with individuals who are ready to try new ideas. It is important to partner with industries 
and to be responsive to their needs. The project leaders suggest that developing an incremental 
plan is best by identifying small steps that can be realistically achieved in certain time periods. 
With experience, these plans can be enhanced and built upon as individuals learn about what 
works best and what can be refined. Some individuals believed that it is easier to begin this type 
of work with the technical fields (such as computer technologies or nursing), where there are 
frequently right and wrong answers. As this project advances, the leaders will learn more about 
the success of their own experiences too. 



Conclusion 

The Community Colleges of Colorado Incumbent Worker Project has the potential to directly 
address key issues facing employers in the state. Through a formal collaboration among 
community colleges and certain corporations, learning experiences will be designed and 
implemented in an effort to improve the performance of new and dislocated workers. Such 
learning experiences should also help these employees to be retained in their positions (or to 
become employable in the workforce) as they become stronger assets to their companies. The 
transportability of these modules should aid other community colleges in different states that may 
be facing similar issues. A national clearinghouse of modules could be exchanged across sectors 
and more directly meet their needs. In the future, formal assessments of participants’ learning 
through multiple methods will actually inform the degree of success with these partnerships. 
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Attachment E-l 



Developer’s Notes: Introduction to StarOfflce™: Using 

StarWriter 5.1 1 



Audience/Context: 

The modules of this course are designed to provide an overview of the basic features of 
StarWriter, the word processing component of StarOfflce™ Version 5.1. The modules are 
suitable for anyone with basic computer skills; familiarity with other word processing 
applications is not necessary. 

The total course is comprised of six modules: 

1 . StarWriter Fundamentals 

2. Formatting a Document 

3. Editing a Document 

4. Styles and Templates 

5. Using Graphics and Other Objects 

6. Converting Files 

Estimated Instructional Delivery Time: 

• The delivery time for Modules 1 - 5 is 75 to 90 minutes. 

• The delivery time for Module 6 is 15 minutes. 

Performance Objectives for these modules: 

1. Identify the basic features of StarOfflce™, navigate its user interface, and create and 
open documents in StarWriter. 

2. Format documents using the full complement of StarWriter features. 

3. Edit and proof documents using StarWriter tools. 

4. Use StarWriter style features to manage formatting 

5. Integrate graphics and other objects into StarWriter documents. 

6. Convert documents between StarOfflce™ and Microsoft Office™. 

Domain and level objectives are drawn from: 

All of the modules and their objectives are drawn from the cognitive domain (understanding 
and application levels) and psychomotor domain (imitation and practice levels). 

Pre-requisite Modules: 

None 



1 Material copied with permission of Alice Bedard Voorhees. 



Baseline Skills: 



Basic computer skills (using the keyboard and the mouse) are required. Experience with 
other word processing applications is not required, but is helpful. 



Classroom Prep: 

Each module includes hands-on exercises for practice with basic features covered. The 
needed files for those exercises are provided in the Using StarWriter folder. That folder 
contains two subfolders: 

• - StarWriter Class - the practice files used for hands-on exercises as well as for 

assessments. 

• Instructor - files that contain completed assessments for the evaluation of student 
assessment exercises. 

It is recommended that the StarWriter Class be copied to the StarOffice Work Folder for 
ease of access during the class. If the files are placed elsewhere, the instructor will have to 
guide students to the appropriate location. 
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Assessment for StarWriter Fundamentals Module 



1) Complete the following: (One point per correct answer) 

StarOffice is an office suite that includes all the essential functions needed for business use, such 
as word processing, spreadsheets, and presentations. 

Name the components of StarOffice used for each listed function: 

• Word processing StarWriter 

• Spreadsheets StarCalc 

• Presentations Starlmpress 

Identify the various elements of the StarOffice workplace, using the illustration on the next page: 
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BEST COPY AVAILABLE 








Complete the following steps. When done, have the instructor check your work. 

a) Open the file “declaration. sdw” in the StarWriter Class folder. 

b) Display the nonprinting characters in the document. 

c) Change the zoom setting to “Page Width”. 

d) Move to the bottom of the document. 

e) Select the last paragraph (beginning with “Note: . . . ”) 

f) Ask the instructor to check your screen. 

Answer : The screenshot below illustrates the correct completion of the steps for #2. Check the 
student’s display for: (One point for each of the following) 

• The file “declaration, sdw” is open 

• Nonprinting characters are turned on. 

• Zoom setting to Page Width (both edges of page are visible on the display). 

• Last paragraph in the document is selected. 
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BEST COPY AVAILABLE 



2) Create and save a new document starting from a blank text document. 

a) Create a new blank document. 

b) Type in the following text in the document: 

Creating a new document is very easy in StarWriter. You can 
start from a blank document, or you can use templates to create 
special documents like fax cover sheets or memos. 

It is helpful to display nonprinting characters while creating or 
editing a document. With nonprinting characters turned on, you 
can easily see how the structure and spacing in the document has 
been created. 

c) Save the document as “practicel” in the StarWriter Class folder. 

d) Close the document. 



Answer: Check the student’s files for the completed file “ practicel.sdw ”. The document should 
correspond to the file “ assessmentl.sdw ” in the “Instructor files” folder. 

(Two points for entering text in document with word wrap; two points for correctly saving 
the document). 



Overall Evaluation: 



Question # 


# Points 


Satisfactory 


1 


7 


4 


2 


4 


3 


3 


4 


4 


Overall 


15 


11 
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APPENDIX F 



Western Governors University 



Introduction 

The purpose of this case study is to explore the competency-based degree programs in place at 
Western Governors University (WGU). WGU is a newly incorporated institution that has dual 
missions for delivering competency-based degrees and brokering of distance-delivered education. 

In fall 1999, the Working Group consultant interviewed five individuals. Interviews lasted from 
30 to 60 minutes or were conducted by email. Two people that work for the initial contracting 
agency that created and developed the first competency-based offerings were interviewed. One 
of them continues as an ongoing member of the WGU Assessment Council; a second is a full- 
time faculty member and department chair at another institution who also sits on the Assessment 
Council. Finally, two WGU administrators — the Senior Academic Officer and the Director of 
Assessment — were interviewed. Additional information gathered and reviewed from the WGU 
Web Site (www.wgu.edu) is from documents on degree development and assessment procedures. 

This case study begins with an overview of Western Governors University followed by an 
explanation of their competency-based degrees. Then assessment strategies used by WGU are 
outlined in addition to a discussion of issues of reliability and validity. Methods for how student 
learning is documented precede a discussion of the data ramifications of WGU competency-based 
degrees. Finally, advantages and challenges faced by WGU are given prior to concluding with 
recommendations for other colleges and universities considering offering competency-based 
degrees. 



Overview of Western Governors University 

As parents and students increasingly worry about degree and course articulation and employers 
question the credibility of college degrees as evidenced by graduates’ usefulness in the 
workforce, governors seek avenues to answer these concerns. In the western region, one answer 
was the creation of Western Governors University. In 1995, governors of most of the western 
states working in conjunction with the Western Governors’ Association agreed to develop an 
accredited postsecondary educational entity that would address these issues. Western Governors 
University, a private nonprofit postsecondary institution, was incorporated in January 1997. In 
the summer of 1999 students were first admitted; at this time, the accreditation process is 
underway. WGU is designed to broker an array of distance-delivered learning opportunities from 
both traditional postsecondary providers and corporate providers to students from across the 
country and the world. In addition, WGU offers competency-based programs of study that are 
designed to recognize knowledge and skills learned through work and life experiences. The 
certificates and degrees, especially those developed initially, support welfare-to-work initiatives 
as well as create opportunities for the geographically dispersed clientele of WGU. Their 
competency-based degrees actively allow students to use past experience toward degree 
completion. It is believed that this aids in accelerating student progress to graduation. 



Competency-Based Educational System 



All WGU degrees are competency-based. The degrees being developed are predominantly at the 
Associate of Applied Science and Associate of Arts level, because information indicates these are 
most needed in the western region of the United States. In addition, master’s degrees in selected 
disciplines are also being created. Baccalaureate degrees are not the primary focus because WGU 
administrators want to focus on specific workforce development needs that call primarily for 
associate’s level degrees and because bachelor’s degrees are being adequately offered by existing 
postsecondary institutions. 

The basic building blocks of WGU degrees are individual competencies — “performance 
descriptions” in WGU parlance. At WGU, they are the smallest unit of describable skill or 
knowledge and are the foundation on which everything else is based. A collection of 
performance descriptions make up a “domain” that corresponds to a body of knowledge roughly 
equivalent to a sequence of courses in a traditional degree program. Similarly, a WGU degree 
comprises several domains (usually six or seven). The initial developers of the WGU degree 
structures found that using competencies allowed for greater flexibility in program and degree 
design, making possible the linking of subject matter in fresh combinations. Subject matter 
“modularized” in this manner allows sharing of topics and competencies between and across 
degrees. Modularization also allows for generic, semi-generic, and specific job-related 
competencies to be collected together and shared in new configurations. For instance, all 
Associate of Applied Science degrees at WGU require the same math domain or collection of 
similar competencies — Applied Quantitative Reasoning — that corresponds to the basic math 
course sequence a student might take at a community college. Similarly, all associate’s degrees 
require the same math domain — Mathematics and Quantitative Reasoning — which contains more 
knowledge and skills than the math domain required for the AAS degrees but has embedded 
within it most of the Applied Quantitative Reasoning domain. (Exhibit 1 has been designed to 
illustrate this property by listing the contents of these two domains side by side.) This use of the 
modular structure supports internal articulation from one type of degree to the next (Associate of 
Applied Science degree to associate’s degree, etc.). Instead of students re-learning subjects from 
earlier in their academic progression, they are able to move to the next level in the subject area 
with less repetition. Another use of the modular structure is that many degree types (AAS, 
associate’s, baccalaureate) may require the same domain. In these ways, WGU believes that 
modularization allows for new methods of mixing and matching knowledge based on new visions 
of how thinking, learning, and work occur in the 21 st century. 

Because seat-time is not the building block of WGU degrees, there are many paths students can 
take to completion of any of the WGU degrees. Unlike most postsecondary degree programs, 
which come with a list of required courses that are to be taken in a suggested order, at WGU the 
number, type, and order of learning opportunities taken by a student depends on what that 
entering student’s academic needs are. Upon entering a degree program, a student is asked to 
work with his or her Advisor-Mentor to catalog and evaluate any prior academic and all work 
history and life experiences. Entering students also take the COMPASS placement examination 
in order to determine their math and English competency levels upon entry. These processes 
establish personal baseline levels of strength and weakness in a variety of academic areas for the 
student, note where improvements are possible, and help the student and Advisor-Mentor identify 
educational offerings that meet the student’s learning style. The student’s Academic Action Plan 
is meant to be adaptive to the student’s needs, but it also includes realistic timelines of projected 
dates for sitting for assessments and progressing toward a credential. When signed by the 
student, this Academic Action Plan constitutes her or his learning contract with WGU (but one 
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Exhibit 1. — Structures of WGU Associate of Applied Science in Electronic Manufacturing Technology 
and Associate of Arts Degrees 1 

Associate of Applied Science Degree 
Electronic Manufacturing Technology 

Applied Quantitative Reasoning Skills 

Arithmetic/ Algebraic Skills 
Statistical/Probability Skills 
Quantitative Problem-Solving Skills 
Quantitative Communication Skills 
Quantitative Technological Skills 



Applied Language and Literacy Skills 
Reading Skills 

Writing/Information-Recording 
Presentation-Related Skills 
Interpersonal Communications Skills 
Basic Information Retrieval Skills 



Basic Work-Related Skills 

Basic Observational Skills 

Learning from Role Models 

Monitoring and Evaluating Own Performance 

Personal Professional Qualities 

Completion of Tasks/Assignments to Client or Supervisor Specifications 
Tools and Techniques for Working with Others 
Personal and Organizational Responsibility and Ethics 
Learning Skills 

Distribution Domain 

Natural Sciences (one course-equivalent outcome examination) 

Social Sciences (one course-equivalent outcome examination) 

Humanities (one course-equivalent outcome examination) 

Specific Applications of General Skills to Electronic Technology Settings 

Electronics-Related Applied Quantitative Reasoning Skills 
Electronics-Related Applied Communication Skills 

Electronics-Related Applied Science and Technology Knowledge and Skills 
Electronics-Related Basic Manufacturing and Workplace Knowledge and Skills 

Electronics Job-Related Competencies and 
Skills 

Basic Knowledge of Electronics and Circuits 
Basic Electronics Skills 

System-Level Setup, Testing and Evaluation Skills 

Associate of Arts Degree 
(transferable degree) 




1 Material copied with permission of Alec Testa, Western Governors University. 
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Exhibit 1. — Structures of WGU Associate of Applied Science in Electronic Manufacturing Technology 
and Associate of Arts Degrees 1 (continued) 

Mathematics and Quantitative Skills 
Numeric and Calculation Skills 
Functions and Algebra Skills 
Geometry and Measurement Skills 

Collegiate Statistical/Probability Skills Quantitative Problem-Solving Skills 
Quantitative Communication Skills 
Quantitative Technological Skills 

Language and Communication Skills 

Collegiate Reading Skills 
Foundational Writing Skills 
Critical and Analytical Writing Skills 
Presentation-Related Skills 
Interpersonal Communication Skills 
Basic Information Retrieval Skills 

Basic Work-Related Skills 

Basic Observational Skills 

Learning from Role Models 

Monitoring and Evaluating Own Performance 

Personal Professional Qualities 

Completion of Tasks/ Assignments to Client or Supervisor Specifications 
Tools and Techniques for Working with Others 
Personal and Organizational Responsibility and Ethics 
Learning Skills 

Collegiate Academic Skills 

Organization Strategies 
Study Skills 
Active Learning Skills 
Academic Advising 
Academic Integrity 

Stress and Procrastination Management 

Distribution Domain 

Fine and Performing Arts (one course-equivalent outcome examination) 

Natural Sciences (two course-equivalent outcome examinations) 

Social Sciences (two course-equivalent outcome examinations) 

Humanities (two course-equivalent outcome examinations) 

History (one course-equivalent outcome examination) 

Collegiate Reasoning and Problem-Solving Skills 
Problem Identification and Clarification 

Identification and Clarification of Stated and Unstated Assumptions 
Analytical Planning and Information-Gathering 
Interpretation and Analysis of Information/Data 
Drawing and Presenting Conclusions 

Communication and Language Skills and Mathematics and Quantitative Skills 
Cross-Disciplinary Issues and Themes 



[To see all WGU degree structures and details, go to the WGU Web Site at www.wgu.edu/wgu/index.html] 



1 Material copied with permission of Alec Testa, Western Governors University. 
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that can be amended). Note that the learning opportunities that a WGU student uses to learn a 
skill or knowledge area may not be traditional college courses. Alternatives include, but are not 
limited to, doing library work, asking to be assigned to particular tasks at their jobs in order to 
learn new skills, or taking a seminar on the subject. In short, WGU is not concerned with where 
students learn something (although a “record of enrollment” is maintained for students taking 
college courses at other institutions, see the discussion below about transcripts and records of 
enrollment); what matters to WGU is whether and how well a student knows something. A 
student’s knowledge and skill level are demonstrated via achievement on assessments. Once a 
student feels confident and prepared, s/he takes the assessment battery required for a particular 
domain. (See assessment discussion below.) 

To determine which competencies should be included in a particular degree, WGU uses a 
rigorous process. Staff consult with individuals working in the particular subject matter area 
from both academe and industry as well as with experts from professional societies and 
accrediting associations. They also gather existing competency statements from a variety of 
sources including the National Skills Standards Board, the Secretary’s Commission on Achieving 
Necessary Skills, the New Standards for Learning Project, ACT’s Work Keys project, course 
syllabi, traditional credit-based program requirements, and professional licensing organizations. 
In addition, confidential internal corporate documents are sometimes provided by working group 
members to help guide structures. Using techniques similar to conducting a job analysis in a 
corporate setting, an initial version of the program structure including explication of the 
certificates and degree as well as the domains and the competencies that are appropriate for each 
domain are created from these sources. 

The structures of the WGU Associate of Applied Science in Electronic Manufacturing 
Technology and the Associate of Arts degrees are given in exhibit 1. As shown, the AAS degree 
is made up of six domains that each have from three to nine subdomains. Six different domains 
comprise the AA degree. In these two degrees there are domains of three types. The first type is 
shared by both degrees (Basic Work-Related Skills). The second type has subdomains that are 
shared by another different domain. For example, the AA Mathematics and Quantitative Skills 
domain contains many of the same subdomains and individual performance descriptions that 
make up the AAS Applied Quantitative Reasoning Skills domain. Similarly, the AA Language 
and Communications Skills domain contains much of the Applied Language and Literacy domain 
from the AAS degree. The third type of domain is unique to the particular credential such as the 
Electronics Job-Related Competencies and Skills domain in the AAS degree and the Collegiate 
Reasoning and Problem-Solving Skills domain in the transferable AA degree. Attachments F-l 
and F-2 contain example performance descriptions for the Applied Language and Literacy Skills 
domain and for the Language and Communication Skills domain. To be awarded a degree, a 
student in the AAS degree program would need to take six assessment batteries corresponding to 
each of the six domains required for the degree. 

In addition to pursuing a nontraditional method for awarding credentials, WGU has taken a 
unique approach to faculty roles. WGU “unbundles” the various activities in which faculty 
members at traditional institutions participate into three primary functions: curriculum 
development, assessment of student progress, and advising. At WGU, Program Councils, the 
Assessment Council, and Advisor-Mentors, respectively, carry out these duties. Curriculum 
development — determination and creation of the competencies — is the responsibility of Program 
Councils that are composed of academic and corporate subject matter experts who oversee 
clusters of credentials in specific areas (for instance, the Information Technology Program 
Council oversees all credentials in the IT area including the Network Administration and the 
Software Application Analysis and Integration Associate of Applied Science degrees). These 
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individuals are employed outside of WGU and are part-time, paid consultants to WGU, which 
allows them to bring an external perspective to the building of degree programs. Student 
assessment is the purview of the WGU Assessment Council, a paid consultant committee of 
nationally recognized experts in the field of assessment. The Assessment Council is charged with 
overseeing the technical aspects of WGU assessment. The final faculty role that has been 
unbundled is that of Advisor-Mentor. These individuals, the primary links between WGU and its 
students, are full-time employees of WGU. As mentioned above, their purpose is to advise 
students of their options and the variety of methods for developing competence. WGU offers no 
instruction and, therefore, employs no teaching faculty. This faculty function, so central in 
postsecondary institutions, is fulfilled via instructors in brokered learning opportunities. 

Responsibility for clusters of WGU degrees lies with WGU Program Councils. Approximately 
nine individuals from collegiate and industrial settings who are subject matter experts sit on each 
Program Council. They give feedback on the staff-prepared degree structures and competencies, 
give suggestions for people to whom one can send the draft structure and competencies for 
additional, broader input, and work with the WGU Assessment Council to choose appropriate 
assessments. Program Council members meet a number of times a year, refining degrees with 
respect to the level and extent of the competencies and what the learner is required to know upon 
graduation. They are ultimately responsible for the content of the degrees and for the quality of 
the students who graduate from the program. Program Councils are equivalent to departmental 
faculty overseeing a traditional credit-based credential. 

To keep the competencies and program structures up to date, the WGU Coordinating Council 
(composed of Program, Assessment, and Education Provider Review Council members and 
administrative staff) has set up a regular program review calendar. Additionally, if the members 
of any Program Council feel that changes are warranted in a credential before the actual program 
review date, then they may also initiate an update. As each degree structure and its accompanying 
competencies are agreed upon and added to the WGU catalog, they are given a “version number.” 
When credential structures are updated, they are assigned a new version number. Students are 
required to fulfill the requirements of the version of a degree in effect when they were admitted. 
If a student has not yet taken a particular assessment and a domain has been updated to a newer 
version, the student may be counseled by his or her Advisor-Mentor that taking the newer version 
would likely enhance job placement. If, however, a student had already passed the assessment 
before a new version was released, the student does not have to go back and take anything new. 



Assessment Strategies 

Although the degree-construction process requires a component analysis of what a graduate must 
do, individual competencies do not constitute the final result. Award of a credential is based on a 
student’s ability to integrate and synthesize the competencies when assessed. At WGU, 
competencies do not stand alone; they guide assessment of student ability. Competencies and 
assessments are therefore fundamentally linked. 

Responsibility for assessment lies with the WGU Assessment Council and the Director of 
Assessment. They oversee assessment policy including guidelines for the delivery of assessments 
and determination of the technical aspects of WGU-sanctioned assessments. Because of the 
importance of integrating the competencies during assessment, an “assessment battery,” a 
collection of assessment instruments and types, rather than a single assessment instrument, is 
often used to assess a domain WGU uses a variety of assessment instruments, including those 
developed by third parties as well as customized assessments built to WGU specifications. 
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When evaluating the assessment options for a particular domain, the WGU Director of 
Assessment and the Assessment Council use several pieces of information. They use charts 
indicating the importance of the various performance descriptions. The appropriate Program 
Council gives a “rating” to each of the competencies for a credential. On a scale of 1 to 3, the 
Program Council members indicate how important the competency is — critical importance, high 
importance, or low importance. This information is used by the Assessment Council to know 
which competencies must be included in the assessment, because not all competencies are 
necessarily specifically assessed. There are drawbacks to this approach, but as one member of 
the Assessment Council said, “WGU may only be measuring criteria imperfectly but it will not be 
granting degrees based on the precise measurement of irrelevant competencies.” Based on 
Program Council input, WGU assessment staff also prepare a “coverage” chart crosswalking the 
performance descriptions with available instruments that show how much of the domain is 
covered by available instruments. Existing published assessments are drawn from Test in Print, 
Mental Measurement Yearbook, etc. The appropriate Program Council guides the Assessment 
Council in term of what methods they envision for assessment of various performance 
descriptions. Finally, the Assessment Council reviews the entire history of the instrument 
provided by the test maker from a reliability and validity standpoint. The Assessment Council 
then makes determinations about the technical properties of the instrument itself. All of this 
information is used by the Assessment Council to make decisions on the use of existing 
instruments for assessing particular domains. 

If the Assessment Council and Program Council find that no instrument adequately addresses the 
domain, they contract out for the creation of new assessment instruments. The more unusual 
domains that require performance tasks in the assessments, such as electronic manufacturing 
technology, have been contracted. WGU is using accepted professional practice for establishing 
the validity and reliability of these new WGU instruments. Standard American English is used, 
and content sensitivity is conducted to eliminate bias. An expert reviews the draft instrument and 
writes a technical critique of it; changes are made accordingly. A pilot test is then conducted 
either in a classroom setting or in a workforce setting using at least 50 people, preferably more if 
possible. Item analysis, including calculation of item difficulty, item/total score correlation, 
internal consistency measures, item means and variances, total and subscore means and variances, 
and item frequency distributions, is done. There are additional revisions based on the results of 
the pilot test and item analysis. Ongoing checks of reliability and validity are scheduled. This 
process is used for portfolios, scoring rubrics, performance assessments, etc. 

When all these data are accumulated, several principles guide the Assessment Council as they 
make their decisions. First, each assessment battery must include some authentic performance 
tasks. Second, different contextual manifestations of the ability must be defined. And third, an 
attempt is made to include different instrument types in the mix constituting the assessment 
battery for a particular domain. 

Based on the data collected and using these principles, the Assessment Council arrives at the 
established assessment battery to be used for each domain. The WGU-acceptable “cut score” for 
competence on any domain is determined using existing information. This information includes 
American Council on Education recommendations for assessments that are already in commercial 
use, cut scores used by other educational and professional groups, as well as those cut scores 
recommended by the test’s publisher. The question of how many distinctions within “competent” 
are appropriate has also been debated at length. Some people argue that it is an either-or 
scenario — either a student is competent or he or she is not. The prevailing view among 
Assessment Council members is that in addition to a minimal cut score reflecting a student’s 
adequacy in a particular domain, there should also be a higher score that distinguishes “mastery.” 
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Thus, for most instruments, two distinct cut scores are used to reflect competent and mastery 
performance. Again, instruments developed for WGU use standard statistical testing and 
norming procedures to determine these cut scores. Attachment F-l contains a description of the 
assessment battery for the Applied Language and Literacy Skills domain from the AAS degree, 
and attachment F-2 contains a description of the assessment battery for the Collegiate Language 
and Communication Skills domain that is part of the Associate of Arts degree. 

Early in the development of WGU, the program councils and the Assessment Council made a 
distinction between those domains that focused on identifiable knowledge and skills and those 
that are attitudes and dispositions. The first are awarded “certificates of mastery” and the second 
are awarded “certificates of completion.” There is only one domain to date that awards a 
“certificate of completion” at this time, that is Basic Work-Related Skills. This difference is an 
acknowledgment that attitude and dispositions are of a different order that other knowledge and 
skills. WGU recognizes these abilities and believes it can sensitize students to socially acceptable 
behavior, but WGU cannot certify that a student always uses these abilities or acts in a sensitive 
manner. Descriptions of the assessment battery for the Applied Language and Literacy Skills 
domain, the Collegiate Reasoning and Problem-Solving Skills domain, and Basic Work- Related 
Skills domain are included in attachments F-3, F-4, and F-5, respectively. 

The reliability and validity of the individual assessment instruments that make up the assessment 
batteries are gathered from the test publishers for existing instruments or are generated in the 
building of WGU-specified instruments. In addition to the Assessment Council reviewing this 
information, the program review process conducted by the WGU Coordinating Council also 
includes consideration of these data in the light of student performance on assessments. A 
method of tracking student performance in the future and asking employers about their abilities is 
planned. Due to limited WGU enrollment at this time, only limited data are available. Data are 
being captured for reliability and validity analyses to be conducted in the future when statistical 
robustness can be ensured. 

It is too early to tell if WGU competencies are valid. Whether or not the competencies are “the 
right ones,” only the market will tell, and again no data are currently available. However, to this 
end, steps have been taken in the development of the competencies to have subject matter experts 
from a variety of situations ratify them up front. Job placement and job success statistics for 
WGU students will be necessary to make decisions on whether the competencies are “correct.” 



Procedures for Documenting Student Learning 

Western Governors University keeps both transcripts and records of enrollment. Transcripts are 
official documents maintained only for degree-seeking students; they include lists of the 
assessments required for the students’ competency-based degree programs, scores for 
assessments and portfolio(s) passed, as well as completion dates for credentials. Assessment 
scores do not expire, but employers and other postsecondary institutions can see how much time 
has elapsed since completion. A Record of Enrollment is maintained for all students who enroll 
for learning opportunities through WGU. This record includes the name of the learning 
opportunity, education provider, and learning outcome or grade. Records of Enrollment are kept 
for the convenience of students and are not official grade transcripts; students must get those 
directly from the education providers. As might be expected at a “virtual” university, most of this 
information is kept electronically in a form that can be printed in hard copy as necessary. 
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Because of the unique degree structure at WGU, academic administrators knew that 
transportability was a key issue. With regard to the transfer of WGU work and credentials to 
other institutions, several articulation agreements have been signed or are in the works with 
individual institutions (to date, Jones International University, Washington State University, 
Regents College, Empire State College, and University of Iowa). Some of these allow transfer 
and articulation of the Associate of Arts degree that was designed to be the first 2 years of a 
liberal arts and sciences baccalaureate degree (such as the transferable AA degree articulating 
with Jones International University’s Bachelor of Arts degree). More states and institutions are 
starting to look at WGU degrees, domains, and competencies with regard to transfer and 
articulation. WGU also articulates with its partner institution, the Open University UK. In 
addition, WGU officials are talking with Utah and Oklahoma to have blanket articulation 
agreements with all public institutions in those states. With regard to the transfer of credit 
achieved at other institutions into WGU, students need only pass the assessment to demonstrate 
their competence. If a student has already passed an assessment elsewhere (as opposed to 
accumulating credit), then the Assessment Council would be asked to review the assessment 
instrument that had been taken. No one has exercised this option yet. 



Data Ramifications 

The change to a competency-based rather than a credit-based degree system has made WGU 
think how student academic information is captured and organized. Transcripts carry domain 
titles rather than course names and assessment scores and completion dates in lieu of course 
grades. Because they started from scratch, there was no need to go back and refashion an existing 
student information system to include competencies and assessment scores. 

In some ways, WGU is better placed to study the reliability and validity of their assessment 
batteries than many traditional institutions. WGU is structuring their student and academic 
information systems from the beginning to be able to conduct statistical analyses. Eventually 
WGU will be able to study patterns of student enrollment in various learning opportunities and 
how well those students performed on subsequent assessment batteries. This information will be 
shared with Advisor-Mentors in order to advise students better on useful learning opportunities. 



Advantages and Challenges Associated with Competency-Based Educational System 

From the beginning, WGU was founded on the concept of competencies. Because they were the 
basis for the institution, the various degree structures as well as the organizational structures of 
councils, staff, and their relationships have as their central purpose the delivery of competency- 
based degrees. In fact, the idea of competency-based education was the reason the traditional 
roles of faculty and administration were completely unpacked and distributed to various staff 
positions, faculty councils, and third-party agencies. This organizational structure allows internal 
checks and balances. Having program councils determine the competencies for a particular 
program without primary concern about the acquisition of appropriate assessment measures helps 
ensure academic integrity. Similarly, other postsecondary and corporate providers supply 
instruction, but competence is determined by a student’s subsequent performance on an 
assessment battery, which provides a check on the quality of instruction. These organizational 
checks and balances were a natural consequence of the a priori commitment to competency-based 
education in the development of WGU. 
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Competencies are the common thread throughout WGU academic offerings. They are the sole 
basis of any WGU credential. Because credentials have a modular structure based on 
competencies, there is additional flexibility that allows lower level certifications to build up to 
full degrees. In addition, the individual competencies or performance descriptions are links to 
learning opportunities offered by external entities. Education providers who choose to list their 
learning opportunities through WGU are given the option to “map” these opportunities to the 
competencies. A course that maps to a WGU competency has that WGU competency as one of 
its learning outcomes. The Education Provider rates their learning opportunity (course) in the 
light of the WGU competency using a scale including “P” or 0 through 5. “P” indicates that the 
competency is a prerequisite for the learning opportunity. If the learning opportunity does not 
address the competency, it is given a 0. If the learning opportunity does cover the competency, 
then a determination must be made regarding to what extent the competency is covered, 1 for low 
coverage and 5 for high coverage. In short, competencies are a primary link throughout the WGU 
academic infrastructure. 

Advantage and Challenge 

WGU is a unique entity in the world of nonprofit postsecondary education, which is both a 
positive and a negative position. Politically it had become apparent to the governors and their 
staffs that the regulatory-push method was no longer sufficiently radical to induce much change 
in postsecondary education, so the decision was made to try a market-pull mechanism. The 
conceptualization of WGU was attractive because it might force existing institutions into doing 
something different and stir up postsecondary education. Some people object to this 
characterization; nevertheless, it was the governors’ intent. While some people feel that WGU 
has had no effect on postsecondary education, governors, state higher education executive 
officers, and presidents of institutions from the western states have noticed changes, particularly 
among regional state college and university systems. Western Governors University’s developers 
point out that despite the difficulties, their biggest success is that they have been able to get the 
institution off the ground at all. 

A major challenge is that the amount of time involved to create competency-based degrees was 
initially underestimated. It took approximately 9 to 12 months to develop competencies for each 
degree, with additional time added to that to determine an appropriate assessment battery. 
(Several degrees were worked on in parallel.) Efforts were made to try to accelerate this timeline 
for each degree and its assessments, but doing a thorough and rigorous job, no matter what 
process was used, took about a year. The competency-based degrees are the heart of the WGU 
mission, and graduates of the programs will be the standard by which the success or failure of 
WGU is measured. Because of these consequences, the planning process cannot be given short 
shrift. 

Another difficulty has been explaining the conceptualization of the competency-based degrees to 
people who have only experienced seat- and credit-based degrees. WGU staff have found that 
people from outside academe (corporate or business people) tend to understand the system more 
quickly than those who worked within postsecondary education already. People want to make 
sense of the unique structures at WGU in terms of what they already know and understand. 
WGU has responded to this by making analogies between traditional structures, functions, and 
roles and those at WGU when possible. 

Another challenge has been the initial assumption that existing assessments would cover the 
necessary competencies. Existing assessments cover only a piece of any given domain. The 
result is that existing assessments must be configured in a patchwork to create the assessment 



O 

ERIC 



144 



155 



batteries. Additionally, difficulty arises in the delivery of assessments that come in a variety of 
formats and styles. A secondary concern here is the legal maneuvering among the various testing 
and assessment groups, which causes some unique problems when delivering assessments from 
competing organizations. 

WGU found that policies based on traditional seat- and time-based measures had to be updated or 
eliminated. The tuition payment process is still time-based, as is the residency requirement that is 
mandated by the federal government — of course, this requirement has little meaning because 
students are not “in residence” and instruction is not offered directly by WGU. The change to 
competency measures has meant that WGU had to submit 14 different waivers on their financial 
aid application. 

Another challenge for WGU has been how to bring expectation in line with reality. The unique 
nature of Western Governors University has made the institution an easy mark for attack. The 
Chronicle of Higher Education has noted the large amount of money from both public (initially) 
and private coffers that has been necessary to bring WGU to limited functioning at this point. 
Misinformation has been fueled by the political rhetoric moving ahead of the actual building of 
the infrastructure. In addition, basic misunderstandings are rife regarding the amount of work 
necessary to create an institution from the ground up— especially one that is not a part of an 
existing institution or system. Responding to these critiques, coupled with the imperative that 
what was being created had to be viable and impeccable, has meant that WGU staff must explain 
the balance of expectation and reality. 



Recommendations for Other Colleges and Universities 

Several recommendations emerge from Western Governors University’s experiences creating 
competency-based degrees. Some have to do with the larger issues of what to do when 
undertaking large-scale change, and others are more specific and have to do with the mechanics 
of using competencies. 

In terms of large-scale change, WGU’s experience with misinformation, expectations exceeding 
reality, and time constraints result in these recommendations. First, interact with as many people 
as possible to distribute information on 1) what the change is, and 2) why it is being undertaken. 
This step may seem trivial, but if the change is radical, it will take multiple discussions to 
sufficiently explain it to others. People who are not involved at the heart of the innovation will 
have a much harder time and need longer to assimilate the information in terms of how they view 
the world. They should be given every opportunity to ask questions and have those questions 
answered. In the building of WGU degrees, questions asked by confused yet interested outsiders 
helped strengthen many of the degree structures. Despite the staffs best efforts, if the change is a 
personal favorite or a hot topic, trustees, presidents, and other academic administrators may talk 
up an innovation before it has had time to sufficiently develop. That happened in the case of 
WGU with the governors’ desire to have the institution functioning almost as soon as they 
thought of it. To bridle expectations in the light of realities such as time, money, and staffing 
constraints, one must share as much information as possible without jeopardizing intellectual 
property. 

With regard to using competencies, several recommendations surface. These include 1) do not 
underestimate the amount of time necessary to do a good job, 2) integrate competencies with 
credit-based systems, and 3) keep in mind assessments from the beginning. The first of these 
seems obvious. Unfortunately, many people underestimate what is involved in competency- 
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based degrees. The faculty overseeing the degree first need to understand competency-based 
degrees in generally the same way. Then the competencies must be compiled, edited, and agreed 
upon. The time involved in these activities is enormous and necessary. Faculty overseeing a 
competency-based degree must understand each competency and why it was included in the 
degree if they are to be able to create learning opportunities and advise student about the 
subtleties of the degree. Discussions of competencies lead to further in-depth investigations, 
particularly among faculty who teach course sequences. Repetitive material is often eliminated 
from subsequent courses, and course materials are altered in terms of the subject matter included. 
Faculty must be given the time to do a good job from the beginning. As faculty become 
accustomed to the process and what is involved in a competency-based degree, the overall 
timeline may shrink somewhat. 

The second recommendation is to integrate competency- and credit-based systems. Since most 
postsecondary institutions are not like WGU and have credits as their academic currency, they 
must keep in mind how they will adapt student and academic information systems and policies to 
allow for both credit and competency attainment. Again, this is not a trivial activity. The student 
information system needs to be investigated to see if it is capable of integrating additional data 
stemming from the assessment of competencies. Can currently unused fields be adapted to carry 
types of assessment, how many attempts at an assessment, scores, subscores if desired, and dates 
of assessment attempts? In addition, policy decisions about how credits and competencies mix in 
terms of student progress and other areas must be made. A policy audit searching for places 
where credits or competencies make a difference should be done. 

Finally, the competency-assessment link is critical. The institution must decide if exams and 
projects created by single faculty members can be used to demonstrate competency or if 
assessments written by groups of faculty and third-party assessments are to be used. Faculty- 
written assessments that undergo the same reliability and validity testing as external third-party 
assessments could be used. However, some faculty might consider this a questioning of their 
ability or an attack on their academic freedom. On the other hand, viable assessments written by 
faculty with corresponding reliability and validity data also become financially viable as third- 
party assessments outside the institution. 



Conclusion 

Western Governors University was created to be a unique postsecondary institution. Along the 
way, they have also fashioned a system for creating competency-based degrees that other 
institutions might want to consider. The jury is still out with regard to how well their system 
works until more data are gathered on student performance both on the assessments and in later 
job success. Until then, it remains an institution to watch. 



Attachment F-l 



Examples of Performance Descriptions 
Applied Language and Literacy Skills 
Copyright Western Governors University 1 



Applied Language and Literacy Skills Domain 
Reading Skills Subdomain (9 performance descriptions) 

Examples: 

Identify statements or conclusions contained within a written passage or document that 
contradict or are not supported by evidence contained within that passage or document. 

Apply the information presented in a written passage to situations not described or not 
completely similar to those described in order to reach a workable conclusion or make an 
appropriate decision. 

Demonstrate the ability to form conceptual images or models from written material that 
contains descriptions of objects or actions. 

Writing/Information-Recording Subdomain (14 performance descriptions) 

Examples: 

Produce written communications that convey information containing several distinct 
points clearly, logically, and without ambiguity. 

Produce written communications that keep to a main point and avoid unnecessary 
information. 

Use a computer word processing program to generate and revise text, including using 
basic keyboarding skills to enter text and numerical information. 

Presentation-Related Skills Subdomain (12 performance descriptions) 

Examples: 

Speak clearly and audibly with a minimum of distractions. 

Select and use appropriate media for presentation to a given audience. 

Adjust style, tone, length, and level of detail presented in an oral presentation to suit the 
needs, experiences, and cultural sensitivities of a particular audience. 



1 Material copied with permission of Alec Testa, Western Governors University. 
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Interpersonal Communications Skills Subdomain (13 performance descriptions) 
Examples: 

Identify changes in tone or emphasis in communications (particularly oral and electronic) 
that result from contextual or cultural differences that amplify meaning beyond the 
message presented. 

Avoid drawing conclusions prematurely before listening to or reading an entire message 
or communication. 

Formulate and ask appropriate questions to clarify points of information or opinion. 

Basic Information Retrieval Skills Subdomain (18 performance descriptions) 

Examples: 

Recognize the variety of approaches to referencing and attribution and their uses, such as 
citations, footnotes, annotations, etc. 

Recognize the varying reliability of sources of information including problems of logic, 
factuality and authority in relation to a given subject or question. 

Use two or more different kinds of documents or displays (e.g., data tables, graphic 
displays, indexed lists of procedures, gauges, and dials) to obtain the answer to a posed 
problem that requires the combined use of several sources. 
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Attachment F-2 



Examples of Performance Descriptions 
Collegiate Language and Communication Skills 
Copyright Western Governors University 2 



Collegiate Language and Communication Skills Domain 
Collegiate Reading Skills Subdomain (19 performance descriptions) 

Examples: 

Identify and describe the specific writing strategies and techniques employed by an 
author in a written passage or text— such as voice, irony, sentence structure, use of 
supporting examples and evidence, use of analogy or verbal imagery, or other stylistic 
features— and evaluate their effectiveness. 

Demonstrate recognition of other voices and/or points of view incorporated into text by 
its author(s). 

Prepare an annotated bibliography outlining the subject matter and arguments of a 
number of texts related to a particular topic or theme. 

Descriptive Writing Subdomain (15 performance descriptions) 

Examples: 

Produce written communications that convey information containing several distinct 
points clearly, logically, and without ambiguity. 

Articulate a process for generating ideas, drafting, and revising written communications. 

Make use of a broad and varied vocabulary in writing and be able to identify or select 
alternative words to convey equivalent meaning. 

Critical and Analytical Writing Skills Subdomain (7 performance descriptions) 

Examples: 

Produce written communications that engage a reader by establishing a context, voice, or 
point of view for the author or that otherwise develop and sustain reader interest. 

Produce a sustained piece of writing intended for a specific audience that analyzes a 
complex idea or posed position from several points of view, compares it to other similar 
positions or arguments, supports it with appropriate examples and references, contrasts it 
to other contradictory positions or arguments, and states a clear conclusion. 

Produce a written analysis of the strengths and weaknesses of a piece of one’s own 
writing by explaining what writing strategies were used, why specific arguments were 
presented as they were, and how the piece might be altered for a different audience. 



2 Material copied with permission of Alec Testa, Western Governors University. 
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Presentation-Related Skills Subdomain (12 performance descriptions) 

[Same as in the Applied Language and Literacy Skills domain.] 
Interpersonal Communications Skills Subdomain (13 performance descriptions) 
[Same as in the Applied Language and Literacy Skills domain.] 

Basic Information Retrieval Skills Subdomain (18 performance descriptions) 
[Same as in the Applied Language and Literacy Skills domain.] 
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Attachment F-3 



Description of Assessment Battery 
Applied Language and Literacy Skills Domain 3 

(Excerpted from document supplied by Western Governors University) 

Requirements for this domain assessment battery include WGU approved assessment activities in 
reading, listening and writing, locating information, English composition, and a performance task. 
The reading exam is a thirty-six-item multiple-choice exam that requires forty minutes to be 
administered. Students review several passages prior to responding to questions. The exam is 
normed for college students at the freshmen and junior level. 

The listening and writing exam is made up of multiple-choice and open response questions. The 
assessment takes forty minutes to complete. Students listen to an audiotape and then respond to 
questions. The exam is normed for adults, and the level of competence mastered by students is 
above the 90 th percentile. The locating information exam is made up of thirty-eight multiple- 
- choice questions, and takes forty-five minutes to administer. The exam is normed for adults, and 
the level of competence mastered by students is above the 90 th percentile. 

The English composition exam is comprised of a fifty-five-question multiple-choice component 
and one written essay question. The multiple section and the essay section each take forty-five 
minutes to administer. The exam is normed for college students, and the level of competence 
required by WGU students meets the recommendations of the American Council of Education. 

The performance task was developed by WGU to measure competence in areas not covered by 
commercially available tests. It covers primarily research and presentation skills. 



3 



Material copied with permission of Alec Testa, Western Governors University. 
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Attachment F-4 



Description of Assessment Battery 
Collegiate Reasoning and Problem-Solving Skills Domain 4 

(excerpted from document supplied by Western Governors University) 

This domain assessment battery is made up of a nationally normed critical thinking examination 
and a WGU-produced series of professional problem-solving activities. The nationally normed 
exam requires short answers and is administered in three parts. The WGU developed activity is 
designed to be conducted with support from the Advisor-Mentor and is a written activity. 



4 Material copied with permission of Alec Testa, Western Governors University. 




152 






1G3 



Attachment F-5 



Description of Assessment Battery 
Basic Work-Related Skills Domain 5 

(excerpted from document supplied by Western Governors University) 

Assessment in this area is formative and not summative. The WGU Program Council intended to 
ensure that students had skills in Basic Observational Skills, Learning from Role Models, 
Monitoring and Evaluating Own Performance, Personal Professional Qualities, Completion of 
Tasks/Assignments to Client or Supervisor Specifications, Tools and Techniques for Working 
with Others, Personal and Organizational Responsibility and Ethics, and Learning Skills. 
Students complete a number of identified assessment activities. The domain assessment battery is 
evaluated for completeness, and the results are shared with the Advisor-Mentor to help inform 
and enrich the advising/mentoring relationship. 



5 Material copied with permission of Alec Testa, Western Governors University. 
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APPENDIX G 



Proficiency-Based Admission Standards System (Oregon) 



Introduction 

This case study focuses on the Proficiency-based Admission Standards System (PASS) in the 
state of Oregon. Their system is one of the more elaborate systems for proficiency-based 
admission in the country and has been in existence long enough to be grappling with how to 
express and share competencies. This latter characteristic was important for studying possible 
data ramifications. In addition, successes and challenges will be highlighted, although the project 
is as yet in its early stages. A more critical perspective will be possible after the new PASS 
system has been used as admissions criteria for several entering classes of students. 



Competency-Based Admissions 

A 1997 survey conducted by the State Higher Education Executive Officers (SHEEO) 
organization identified 1 1 states (approximately 20 percent of the nation) contemplating or 
instituting competency-based admission systems (Russell, 1998). Many of these states 
(California, Georgia, Iowa, Kansas, Minnesota, and New York) are only in the embryonic stages 
of development, gathering preliminary information for review. Of the five remaining states, 
Colorado and Wisconsin are focusing their efforts on creating dual admission systems — using 
competencies alongside the existing traditional admissions criteria. Wisconsin is further along in 
its development currently investigating the effects of their dual system on the various groups of 
students who have been admitted by the two systems (University of Wisconsin System, n.d.). 
Finally, three states — Maryland, Oregon, and Washington — are planning eventually to have a 
single state system for college admission based solely on competencies. Maryland’s reason for 
doing so is to ease transitions in the K-16 continuum. The impetus for adopting competency- 
based admissions systems in Oregon and Washington is to build on and integrate with K-12 
systems that are increasingly competency-driven. 



Site Visit 

The PASS offices are located on the campus of the University of Oregon in Eugene. Five people 
were interviewed in person or via telephone. Additional information and reading materials were 
gathered both in person, from the Web, and in subsequent telephone conversations. Interviews 
were conducted with three staff members — the Executive Director, the Assessment Director, and 
the Evaluator — and two people external to the project but well acquainted with it — a former 
university admissions director and an administrator from the Oregon University System. Three 
individuals were advocates of the new system, while the other two had reservations about PASS. 
Interviews focused primarily on the consequences this project would have within the 
postsecondary sector and how it would effect the use of data as it applies to issues of reliability, 
validity, and transportability. 



History 



The origins of PASS begin in 1991 with Oregon’s K-12 reform efforts. In 1991 and 1995, the 
legislature adopted the use of the Certificate of Initial Mastery (CIM) at the 10 th grade level and 
the Certificate of Advanced Mastery (CAM) at the ,12 th grade level. The CIM and CAM both 
focus on competencies or “proficiencies” necessary for student achievement. CIM requires 
academic achievement in English, math, science, social sciences, arts, and a second language. 
CAM requires achievement in the same academic areas plus generic career-oriented 
competencies in “personal management, problem solving, teamwork, communication, workplace 
systems, career development and employment foundations” as well as application of academic 
knowledge in a work context (Oregon Department of Education, n.d.). In response to these 
changes at the secondary level, the Oregon University System (OUS) in 1994 adopted a 
proficiency-based method for admission to colleges and universities in its system; note that there 
are no statutory requirements for PASS. PASS is to be phased in over a number of years with full 
implementation expected with the entering class of 2005. 



The PASS Project 

OUS chose to go to a proficiency-based system “to increase the likelihood that students who are 
admitted to college will be able to succeed and to graduate in a timely fashion” (PASS handout, 
“Overview of Changes in Oregon University System College Admissions Expectations,” April 
1999). PASS is focused exclusively on academic subjects and whether a student is prepared for 
academic work in higher education; it is not about workplace or employment competencies. 
PASS advocates point out that another reason for moving to proficiency (and competency-based) 
models is the current dissatisfaction with the reliability of grade point averages as proxies for 
student learning; they believe that in the long run, proficiencies will be better measures. In 
addition, administrators would like to use PASS proficiencies and scoring in placement decisions 
as well. Finally, another suggested future outcome of PASS is that the Associate of Arts Oregon 
Transfer (AAOT ) degree will eventually be defined by proficiencies rather than the credits now 
used, removing obstacles to seamless transitions along the path to a baccalaureate degree. 

When fully implemented, PASS will require students to demonstrate proficiency in six subject 
areas — math, English, social science, science, visual and performing arts, and second languages. 
For each of these subject areas there are 4 to 7 individual proficiencies for a grand total of 33 
proficiencies (plus 1 recommended proficiency). PASS proficiencies are “what it is we want 
students to know and be able to do; there is both a content piece plus an intellectual process side 
to the proficiencies” (interviewee quote). One positive outcome of this explicit stating of 
proficiency criteria is that high school students appreciate knowing exactly what is expected of 
them (PASS, n.d.). The proficiencies are being phased in as admission requirements over 4 years. 
English and math will be required in 2001, science is added in 2002, social science in 2003, 
visual and performing arts in 2004, with second languages being the last added as a requirement 
in 2005. Once PASS is fully implemented in 2005, the old admission system will effectively be 
phased out. (See the complete list of proficiencies at the end of this document.) 

Concerns of parents, students, and admission officers center on transfer and articulation across 
postsecondary systems. These have been addressed by working with major out-of-state 
institutional receivers and providers of students. Agreements in principle have been reached with 
the University of California and California State University systems, the University of 
Washington, and private universities such as Stanford have provided letters of support to PASS. 
Additional concerns center on the award of financial aid and scholarship monies that are based on 
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grade point averages and credits and NCAA requirements. Steps are being taken to address these 
issues. 



Proficiency Assessment and Scoring 

Each of the 33 PASS proficiencies is “scored” into one of the following categories: 

• No evidence of proficiency; 

• Working toward proficiency (evidence collected indicates that the proficiency has 
been attempted but does not indicate proficiency); 

• Partial proficiency (a student’s collection or test score suggests proficiency but is not 
enough to convince there is proficiency); 

• Proficient; 

• Highly proficient (more and better at doing the proficiencies than required); and 

• Exemplary or unique proficiency (work exemplifies the proficiency, or is so unique it 
needs to be recognized by university admissions whether it meets the proficiency or 
not). 



Students may present assessment tasks, collections of evidence, or a combination of assessment 
tasks and collections to establish their levels of achievement for the proficiencies. Acceptable 
assessments include state multiple-choice tests (which include national examinations such as AP, 
CLEP, ACT, and SAT II) or state on-demand assessment tasks that require response to an open- 
ended question that is then distributed to graders. Alternatively, students may achieve 
proficiency using academic work collections. Student work collections are “scored” by the 
students’ teachers in lieu of high school grades. This mechanism allows teachers to incorporate 
the scoring of proficiencies into their existing grading process and will one day replace the 
traditional grading process. Because of this substitution, there are no additional costs to pay 
scorers and the scoring workload is distributed across secondary school teachers. Some schools 
and districts may one day investigate doing scoring collectively, but that decision is internal to the 
individual school or district and in the future. 

Teachers initially judge student collections of work on two properties. First, is the collection 
sufficient, and second, is it proficient. “Sufficiency” in this context means that the collection 
represents the student’s performances and “adequately represents the proficiency [to be 
demonstrated] and creates a fair opportunity for the student to demonstrate his or her knowledge 
and skills through a variety of tasks” (PASS Notes, May/June 1999, p.2). One respondent used 
the legal analogy, “is there enough evidence to indict?” 

“Proficiency” in this context means that the collection is indicative of the student’s mastery of the 
necessary knowledge and skills. Each piece contained in the collection is rated. Note, the 
number of pieces in a collection can vary, and the collection is not supposed to contain enough 
pieces to allow a one-to-one check-off of the proficiencies because proficiencies, like 
competencies, are not enacted in isolation but are integrated in the work. Students choose, in 
consultation with their teachers, what to include in the collection. This allows students the 
freedom to “put together lots of different combinations of evidence — this is one indication of the 
flexibility in the system” according to of one respondent. Finally, the teacher (or teachers) who is 
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judging the collection categorizes the student work collection for each proficiency for which it 
was submitted. 

High school teachers are provided with criteria and guidelines and are invited to “scoring events” 
and “verification institutes” in order to learn how to score and judge collections and then verify 
one another’s judgments. These institutes are sponsored and paid for by PASS using grant 
money. The need for these institutes will diminish as exemplars and scoring guides are created 
and teachers become comfortable with the meaning of “proficient” in many settings. Criteria for 
each of the proficiencies have been published and guide the work of teachers as they categorize 
student work collections. An example of a criterion for the English proficiency “interpret literary 
works” is “interpretation and use of textual evidence,” described as “students demonstrate the 
ability to use textual evidence to develop and support an interpretation of a literary work” 
(example from http://pass-ous.uoregon.edu/standards/english/eng_summary.html). “Calibration 
institutes” are also being held with university faculty to allow them to understand the scoring and 
verification procedures and see examples of high school student work for themselves. 

In the early stages of the system, cut scores are based on empirical studies and advice given to the 
PASS staff. These initial cut scores will be used to establish a baseline that will then be adjusted 
based on validity studies and studies to determine the predictiveness of the scores. Cut scores are 
set for the entire Oregon University System, in part because the philosophy of PASS is that as a 
state system a certain level of achievement is required of students, and if students achieve at that 
level of proficiency then they are capable of doing well at any university in the system. Each 
university, or major, is free to establish more stringent proficiency cut scores, but OUS will have 
only one. It is hoped that by focusing on setting a single standard for student performance, (1) the 
extraneous information that filters in with grades will be eliminated, and (2) the process will be 
less competitive. 

There are no student portfolios nor are student work collections forwarded to universities. 
Instead, PASS has created a “proficiency transcript” (see example in attachment G-l), which 
university admissions offices can use quickly and at a glance. Each subject area lists the number 
of proficiencies with space for the proficiency score, the assessment method used, and the date 
the assessment was verified. In addition, the transcript has space for recording the dates CIM 
proficiencies were awarded. 

Work has just been completed bringing the CIM, CAM, and PASS proficiencies into alignment. 
Designed for different functions — CIM and CAM are proficiencies for “getting out” of high 
school and PASS proficiencies are for “getting into” college — they are nevertheless linked. While 
not a linear progression, both CIM and CAM contain proficiencies that will fulfill PASS needs. 

Innovative methods for college admission must be rigorous with regard to the reliability and 
validity of the measures and processes used, and steps are being taken to fulfill this need. 
Multiple measures are available for determining proficiency, cross-scoring of student collections 
occurs by multiple teachers, and researchers gather data on pilot students and from early scoring 
and verification events to study the predictive validity of the proficiency scores given to students. 
One day these predictive validity studies will yield information useful to students who have taken 
time off between high school and college as they choose the best courses to take as they re-enter 
their academic studies. In addition, an American hybrid of “moderation panels” used in Australia 
will be looking for systematic over- and under-rating by teachers and will make adjustments 
accordingly. The PASS Assessment Coordinator is planning to investigate patterns of scoring for 
teachers used as raters and for groups of teachers. 
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Policy Issues 



Oregon has some remaining policy issues to be confronted in the next few years, including 
agreeing on levels of achievement for exhibited proficiency that reflect desired incoming 
freshman ability levels. They know also that once these standards are in place, there must be 
support systems created for students who fail to achieve proficiency. In addition, OUS and its 
institutions should be prepared for any decreases in funding that might occur if a substantial 
portion of students do not exhibit proficiency and their subsequent entering enrollments decline. 
However, Oregon’s enrollment-driven funding formula is constructed such that retaining more 
students, even if enrollments do decrease, could result in a funding increase. 



Successes and Challenges 

At this point in the development of PASS, it is easier to see the challenges that still must be faced 
than to identify the successes. Successes are clearer after a project has been given time to be 
implemented and mature. Nevertheless, initial successes might include the administrative 
acceptance of a proficiency-based system and the seeming acceptance by faculty; In addition, 
PASS has been given ample development time. This characteristic is critical to decisionmaking. 
Because PASS was not created in response to a crisis, adequate time has been allowed for 
reasoned decisions to be made. Another success is that high school students and teachers like the 
explicit nature of the proficiencies and knowing what is required of them (PASS, n.d.). Finally, 
the efforts to ensure articulation with university systems outside OUS and the state have been 
impressive. 

With regard to challenges, many already exist, and in the coming months others will emerge 
during implementation. The relative silence of faculty with regard to proficiencies now might not 
be tacit approval, it may just mean that proficiencies have not yet directly affected faculty work. 
Their buy-in is critical to the success of the project. One difficulty has already become apparent 
in the different standards that groups of faculty apply to the various proficiency areas — math 
faculty think that a more advanced level of math is required, while natural sciences faculty think 
that a more moderate level of math understanding is enough. Another challenge and perhaps 
misstep taken by the PASS coordinators is the tepid involvement of sitting admission directors. 
Some of them feel that their concerns regarding NCAA requirements for scholar-athletes and 
financial aid have not been adequately addressed. In practice, these issues will need to be quickly 
resolved to avoid penalizing students as the system becomes functional. Finally, as with any 
proficiency or competency-based system, the proof will be in the assessment of the proficiencies, 
the consistency with which they are judged, and how students admitted using the new method 
fare within the Oregon University System. 



Conclusion 

The PASS system is an example of the use of competencies — proficiencies — for the transition 
from secondary school to universities. A great deal of thought has gone into determining how the 
system will work. PASS’S progress and data reports will need to be monitored as the system is 
phased in to see how they are faring with regard to acceptance of and student success under the 
new system. 
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Attachment G-l 



PASS PROFICIENCIES 1 

(Adapted from Proficiency Standards: Summary Charts of Criteria for All Content Areas, 
1998-99, PASS, Oregon University System, 

Office of the Chancellor, Eugene, OR^ 



English Proficiency Areas 

Read From a Variety of Literary Genres And Periods 

Read and respond to a broad selection of literature from a variety of historical 
periods, cultures, literary perspectives, and genres, including poetry, novels, short 
stories, essays and drama; understand the characteristics of literary genres, 
periods, and movements. 

Interpret Literary Works 

Analyze literary forms, elements, devices, and themes to interpret and critique 
literary texts, performances, and media. 

Analyze Relationships of the Humanities and Human/Social Experience 

Explain how the humanities reflect, influence, and comment upon human 
experiences and societal assumptions, traditions, structures, and changes. 

Conduct Inquiry and Research 

Conduct inquiry and research, using a variety of primary and secondary sources 
and informational resources to investigate questions and topics, gather and 
synthesize information, and create and communicate knowledge. 

Communicate in Oral, Visual and Written Forms 

Use oral, visual, written and multi-media communication forms to convey 
information and ideas for a variety of purposes, audiences, and contexts. 

Write for Varied Purposes 

Write to discover and convey meaning, using effective processes to produce 
writing that is thoughtful, fluent, organized, coherent, and clear. 

Math Proficiency Areas 

Perform Algebraic Operations 

Use algebraic operations and mathematical expressions to solve equations and 
inequalities including, but not limited to, exponentials and logarithms. 

Use Functions to Understand Mathematical Relationships 

Use patterns and functions to represent relationships between variables and to 
solve problems; interpret and understand the connections among symbolic, 
graphic, and tabular representations of functions. 



1 Material copied with permission of Christine Tell, Oregon University System. PASS standards are updated annually. 
See PASS Web Site: http://www.ous.edu/pass/ 
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Use Geometric Concepts and Models 

Represent and solve problems with two- and three-dimensional geometric 
models, properties of figures, analytic geometry, and trigonometry. 

Use Probability and Statistics to Collect and Study Data 

Use probability and statistics in the study of various disciplines, situations, and 
problems; understand and apply valid statistical methods and measures of central 
tendency, variability, and correlation in the collection, organization, analysis, and 
interpretation of data. 

Estimate and Compute 

Use computation, estimation, and mathematical properties to solve problems; use 
estimation to check the reasonableness of results, including those obtained by 
technology. 

Solve Mathematical Problems 

Apply mathematical problem-solving strategies to problems from within and 
outside mathematics; devise, implement, evaluate, and communicate processes 
and solutions; select and use appropriate models, operations, and technologies. 

Reason Mathematically 

Formulate and test mathematical conjectures (i.e., make generalizations from 
observations); draw logical conclusions from given or known information; follow 
and judge the validity of mathematical arguments and proofs. 

Science Proficiency Areas 

Know Fundamental Concepts of the Sciences 

Know and apply fundamental and unifying concepts from the physical, life, and 
Earth and space sciences, demonstrating general scientific literacy. 

Understand, Use, and Investigate a Domain of the Sciences 

Understand, use, and investigate essential concepts, principles, theories, 
relationships, and experimental processes in a domain of the sciences, exhibiting 
a specialized scientific competency. 

Analyze Scientific Knowledge, Theories, and Research 

Analyze and evaluate scientific information and claims to understand the nature 
of scientific knowledge, the context in which scientific theories and concepts 
develop, and the implications of scientific research for society. 
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Design and Conduct Scientific Investigations 

Design and conduct experiments using principles of scientific inquiry, 
investigative processes of the sciences, scientific instruments and technology. 
Collect and analyze data, critique experimental designs, and communicate 
scientific problems, results, and arguments. 

Social Science Proficiency Areas 

Know and Use Geographic Information 

Know and use geographic information, concepts, and skills to understand and 
, analyze historical, social, economic, political, cultural, and environmental issues. 

Understand Patters of Human History 

Exhibit knowledge of the chronological flow of human history; identify major 
themes of historical change in prehistoric through contemporary periods. 

Understand United States History 

Understand significant eras, concepts, people, events, and relationships in U.S. 
history. Employ historical thinking and inquiry to understand events, issues, 
developments, relationships, and perspectives of history, and to interpret current 
trends and issues. 

Understand Structures and Systems of United States Government 

Understand the principles, purposes, structures, and functions of government in 
the United States: its philosophical basis and historical evolution; the structure of 
power, authority, and governance; the relationship of the states to the federal 
government; the Constitution and Bill of Rights; the dynamics of conflicting 
rights and interests in the American political system; the role and responsibilities 
of citizenship; and patterns of democratic participation in American politics. 
Compare other forms of government and political systems to those found in the 
United States. 

Analyze Economic Systems 

Analyze the structure and functioning of economic systems, their relationship to 
national and international political, social and geographic systems, and the 
conditions that influence the development of such systems. 

Conduct Social Science Analysis 

Research and analyze complex societal issues, phenomena, and events, posing 
and communicating potential responses. Use the varied perspectives of the social 
sciences, varied data sources, and technology in social science inquiry and 
analysis. 

(Encouraged) Examine Aspects of Human Behavior 

Examine dimensions of individual and group behavior and the effects of culture 
and ethnic diversity within and among societies, using information from the 
behavioral and social sciences. 
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Visual and Performing Arts 

Understand Elements, Principles, and Processes in the Arts 

Understand the ways in which artists use elements, materials, technologies, 
artistic processes and organizational principles in similar and distinctive ways in 
various art forms and disciplines. 

Understand the Role of the Arts in Society 

Understand the role the arts play in society and the ways in which the arts 
empower people to create images, artifacts, performances, and structures that 
manifest their beliefs, knowledge, social relationships, values, and skills. 

Interpret Art from Various Cultures and Historical Periods 

Interpret works of art from various historical periods, cultures, and peoples, 
analyzing the context in which they were created, the characteristics of the 
works, and the range of possible interpretations. 

Appreciate the Arts 

Understand the significance and potential contribution of the arts to the quality of 
one’s life. 

Exhibit Skill in One Discipline of the Arts: Drama, Visual Arts, Music, Dance 

Exhibit skill and understanding in at least one form of the arts, defining and 
responding to artistic problems with insight, technical ability, and quality. 

Analyze and Critique Artistic Works 

Analyze and evaluate works of art and performances from functional, 
structural/formal, historical, and cultural perspectives, using defensible criteria 
and communicating effectively through writing, speaking, and expressive media. 

Second Languages 

Oral Communication 

Use spoken language to communicate the content of your message to others and 
to comprehend the content of others’ messages to you. 

Reading 

Comprehend general meanings and specific details contained in written texts. 

Writing 

Convey content through legible and comprehensible text. 



Culture 

Use appropriate verbal and non-verbal expressions reflecting knowledge of the 
culture(s) in which a language is spoken. 
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APPENDIX H 



Ford Motor Company 



Introduction 

In the spring 2000 semester, five individuals from the Human Resources unit in the Ford World 
Company Headquarters in Dearborn, Michigan, were interviewed about their competency-based 
selection and training system. Each individual interview during the site visit lasted between 50 to 
60 minutes. Numerous company documents that provided additional insights about their 
competency-based model were reviewed by the consultant. The purpose of this case study was to 
explore the competency-based system at Ford Motor Company with a particular focus on the 
development of competencies, how they are assessed, and data ramifications. 



The Setting 

The Ford Motor Company produces cars and trucks. The- company and its subsidiaries also 
engage in other businesses, including manufacturing automotive components and systems and 
financing and renting vehicles and equipment. Its automotive vehicle brands include Ford, 
Mercury, Lincoln, Volvo, Jaguar, and Aston Martin. Ford owns a portion of Mazda Motor 
Corporation. The Company also completed purchase of the AB Volvo’s worldwide passenger car 
business. 

There are six different leadership levels for salaried employees. The top level consists of about 
40 officers, the next level includes approximately 200 executives, followed by 2,000 mid-level 
managers. There are about 40,000 entry-level managers. Ford Motor Company has structured 
“global excellence centers” where employees may work in manufacturing, purchasing, marketing, 
finance, business development, process leadership, or product development. 



Identifying Important Outcomes 

Ford Motor Company uses competencies in their selection, training, and promotion of global 
salaried employees. Ford defines competency as “an agreed upon set of knowledge, skills, 
abilities, experiences, values, or personal characteristics that we believe are necessary to run the 
business and achieve our business objectives” (Ford Motor Company, 2000, p. 4). 

The competency model that Ford developed was based upon a formal review of existing models 
in other companies. Both an internal and external benchmarking process was followed by 
comparing themselves with other known companies that had the “best-in-class” selection 
procedures. Companies such as Philip Morris, Texas Instruments, and British Airways were 
contacted so that Ford could review and examine their practices. They performed an analysis of 
effective behaviors particularly for managers. They found that structured assessments (paper- 
and-pencil instruments and job simulations) were common across these companies. 

The Ford Motor Company then constructed a composite of competencies that were reviewed by 
executives and then by the Executive Personnel Committee who had to formally approve them. 
The specific functions and levels of adaptation were left up to individual units to determine their 
specific applications. The initial focus and use of competencies for global salaried employees 
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selection were in four segments: (1) engineering, (2) human resources, (3) production and 
supervision, and (4) marketing and sales. 

One main reason that Ford wanted to move toward using competencies in employee selection was 
so that they could provide the company with a competitive advantage. They also expected that 
using competencies would lead to the following strengths: 

• Better prediction of job performance; 

• Measure relevant skills and behaviors; 

• Promote a global process; 

• Be more flexible; and 

• Be easier to administer. 



The competencies are currently clustered into three main areas that are considered to be important 
leadership behaviors: (1) “heart,” (2) “relationship,” and (3) “know-how.” Each cluster is briefly 
defined by a few key terms. Then, there are four subcategories under each major category that 
further define the specific competencies. The Ford Leadership Behaviors are outlined below 
(Ford, 2000). 1 



“HEART” 

• Integrity (Stays faithful to promises) 

■ Keeps one’s work and commitments, exercises principled judgments, and accepts 
responsibility for actions; 

■ Lives Ford Motor Company’s values and demonstrates pride in our heritage; and 

■ Accepts accountability for environmental and community responsibilities. 



• Courage (Willingness to act) 

■ Demonstrates reasoned fearlessness in taking risks that transform and grow the business; 

■ Unafraid to act quickly for Ford Motor Company and consumer interests; and 

■ Assumes personal responsibility for “doing the right thing.” 

• Durability (Goes the extra kilometer) 

■ Maintains originality and creativity in staying the course to achieve agreed upon 
objectives; 

■ Show tenacity and boldness in securing and using resources; and 

■ Maintains inspiration, focus, intensity, and persistence, even under adversity. 



1 Material copied with permission of John M. Rauschenbarger, Ford Motor Company. 
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• Drive for Results (Show passion for accomplishment) 

■ Determined to achieve challenging, innovative objectives and accepts personal 
responsibility for accomplishing them; 

■ Prioritizes resources, inspire performance, enhances productivity, and measures 
outcomes; and 

■ Negotiates agreements that move the business forward. 

“RELATIONSHIP” 

• Teamwork (Gets on Board) 

■ Holds self and other personally accountable for engaging in team-oriented behavior; 

■ Values different ideas, points of view, and backgrounds; and 

■ Demonstrates respect and trust in fellow employees. 

• People Development (Cares, Nurtures, and Safeguards) 

■ Values and confidently promotes a diverse workforce; 

■ Believes and practices continuous learning and develops self and others; and 

■ Safeguards employees’ personal well-being and ability to honor obligations to 
community and family. 

• Desire to Serve (Connects with the Consumer) 

■ Strives to develop a deep knowledge of consumer values; 

■ Respects diversity of consumer needs, desires, and aspirations; and 

■ Committed to surprise and delight the consumer. 

• Communication (Stays in Touch) 

■ Communicates/teaches Ford Motor Company’s core purpose and values; 

B Listens completely; and 

■ Communicates concisely with conviction and cultural sensitivity. 
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“KNOW-HOW” 



• Business Acumen (Shows Business Savvy) 

■ Knows Ford Motor Company’s global business operations and the global business 
context in which the company operates; 

■ Knows how to achieve consumer quality and profit outcomes in a global environment; 
and 

■ Demonstrates functional and technical expertise. 

• Systematic Thinking (Sees Consequences and Potential) 

■ Thinks cross-functionally about ideas that impact the business; 

■ Boldly pursues ways to improve processes and incorporate new ideas; and 

■ Inspires systematic change efforts that make a difference. 



• Innovation (Ingenious in Applying Expertise) 

■ Demonstrates intensity and focus in discovering new ways of satisfying customers; 

■ Discovers, empathizes with, and responds quickly to consumer dreams and aspirations; 
and 

B Committed to surprise and delight the consumer. 

• Quality Methods (Delivers Excellence) 

■ Shows passion for achieving quality and strives to continuously improve our business and 
behaviors; 

■ Measures and monitors quality on an ongoing basis; and 

■ Demonstrates commitment to meet customer quality requirements. 

In addition to these leadership behaviors, Ford also contracted with external consultants to 
perform job analyses for the main clusters of workgroups (such as human resources and finance) 
to determine the characteristics and dimensions of effective performance. Industrial 
organizational psychologists served as the external researchers who studied certain managers for 
3 to 5 years at Ford and identified effective performance. Once the external consultants 
completed the job analyses, then they also developed competencies linked to these clusters of 
workgroups. The results from these job analyses were then mapped onto the Ford leadership 
behaviors. All competencies are fairly steady over time. Even with changes in technology, the 
results from the job analyses are steady for several years. 

Within the Human Resource System Infrastructure, these competencies guide processes at several 
points in time. First, competencies are used as a basis to screen individuals who are recruited for 
positions. Second, competencies are formally assessed to identify who will ultimately be selected 
or hired to fulfill specific positions. Competencies are used to evaluate the performance of 
management. They also guide the design of training seminars and workshops. Finally, 
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competencies are used to create development plans for leaders who aspire to make a lateral move 
within the company or seek a promotion. 



Assessment Strategies at Ford Motor Company 

Assessment at Ford begins with the recruitment of potential employees to be considered for 
professional positions. During the initial stage or pre-screening (usually conducted on college 
campuses), prospective applicants supply a resume that outlines their educational background and 
their work experiences as well as responsibilities. They also complete a structured interview that 
lasts about 30 minutes and a Situations Inventory (SI) (that takes about 35-45 minutes). The SI is 
a paper-and-pencil assessment that contains various scenarios that an applicant is likely to 
encounter on the job. The applicants are required to rate the desirability of various solutions. 
The specific situations and behavioral options were created by Ford employees in each job 
family. The SI is structured for engineering, production supervision, human resources, and 
marketing and sales. 

Approximately 2,000 to 3,000 candidates are assessed each year. For candidates who make it to 
the next step, there is a formal assessment at Ford that involves two major stages. First, there is a 
structured 1-hour interview with each candidate. Then, all candidates participate in a job 
simulation that lasts 3 hours and 10 minutes. Ultimately, about half of the assessed candidates are 
hired. 

The job simulation usually consists of several components. Candidates are randomly divided and 
placed into groups consisting of 12 members who are all applicants for positions at Ford. All 
assessments are developed by Ford employees and tailored to the specific job function as well as 
the competencies. Each team of individuals progresses through multiple assessments. One 
formal assessment is a case of an actual business problem. Teams of individuals write about their 
recommendations in terms of solutions to the particular problem. A second assessment is a 
stressful role-play situation. For example, an individual will need to respond to an irate customer 
or car dealer. Another assessment consists of an individual’s presentation about solutions to a 
particular problem. The third assessment is a group discussion about a problem. Typically, these 
structured assessments are designed to address important competencies for about 2 years. Then 
they are modified as needed. Across these assessments, each assessor uses a checklist of 
behavioral indicators and rates each candidate’s performance. The scale ranges from ineffective 
(with a score of “1”) to very effective (with a score of “9”). 

Professional staff in an assessment center are trained for 2 days to learn how to evaluate a 
candidate’s performance in the job simulation. Examples of effective performance serve as 
anchors that illustrate performances at each of the three different levels: very effective (scores 
ranging from 7 to 9), effective (scores ranging from 4 to 6), and ineffective (scores ranging from 
1 to 3). Through this training process, assessors reach greater consistencies in their judgments 
about candidate’s performances. Once professional staff complete their 2-day training sessions, 
they become certified assessors. 

At least two assessors evaluate each simulation. A more highly trained coordinator supervises 
each pair of assessors. This coordinator is responsible for reviewing their assessments and then 
working with them to resolve any major differences in their ratings. They discuss reasons for 
significantly different judgments and then reach a consensus about a common specific rating. 
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There are no cutoff scores required to be hired at Ford. However, candidates who scored at the 
very ineffective level across all simulations and assessments would not be hired. Leaders at Ford 
believe that some skills can be strengthened once employees are hired. For example, some 
candidates may have weak writing skills, but through training at Ford, those skills can be 
improved. 

The Human Resource staff believe their training of assessors strengthens the reliability of their 
results. In addition, they are making progress toward predictive validity, since they use actual 
problems that are encountered at Ford as the major assessments. The assessment system has not 
been in place long enough to reach strong predictive validity, but they view this as being 
accomplished with more experience over time. 

These types of assessments are believed to ultimately improve selection decisions at Ford and to 
send a positive message about the company as well. In addition, since employees are assessed by 
formal methods, it increases objectivity and gives candidates better information upon which to 
decide if they want to work at Ford. Finally, the company gets an early glimpse into the skills 
and knowledge of each individual that allows them to know where the future employee needs 
further development once he/she is hired. 

The costs associated with assessment are not issues for Ford. The belief is the more data that can 
be gathered about employee performance, the greater likelihood that an individual can work on 
weaknesses. 



Assessment of Ford Leadership Behaviors: The 360 Degree Leadership Assessment Process 

There are three major roles or groups of individuals who are directly involved in this particular 
assessment process. First, those being rated are receiving feedback on their leadership behaviors. 
Ratees must identify individuals whom they would like to evaluate their performance. Their list 
of nominees must include a direct manager and one matrix manager (if appropriate), six work 
partners, and up to six direct and/or matrix reports as raters, and individuals who have observed 
their leadership behaviors. Second, the raters are individuals who formally evaluate and provide 
feedback on leadership behaviors of particular individuals. Third, the manager reviews the direct 
results and integrates this information into the performance management process. 

Ratees discuss their nominations with their direct managers and must get the managers’ formal 
approval. Managers must evaluate whether the group of proposed raters are in the best position to 
provide knowledgeable feedback and unbiased feedback that is meaningful and useful. 

All raters provide feedback on the Ford Leadership Behaviors via the Ford intranet. Raters are 
asked to provide a thoughtful assessment and constructive comments that will help their 
colleagues make important decisions about their leadership behaviors. The entire assessment is 
completed on line by the raters. Each individual is provided with scaled behavioral examples, 
used to help each rater make an informed judgment about an employee’s leadership. The 
beginning instructions for this particular assessment include additional guidelines. Raters are 
asked to base their ratings on behaviors that they have observed during the past 12 months (not an 
old behavior or behaviors that they think someone might demonstrate in the future). They are to 
consider nonverbal behaviors too, particularly when there are significant differences between a 
person’s words and actions. Raters are cautioned to make sure that they use the entire rating scale 
when appropriate. This caution is advised since some raters may be inclined to keep all ratings in 
the middle or at one end of the spectrum. The scale ranges from 1, ineffective, to 3, somewhat 
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effective, to 5, effective, to 7, very effective, to 9, role model. Following is one example of the 
key illustrations for one particular competency. 2 



Courage: The Willingness to Act 

Ineffective Behaviors 

• Tells others what they want to hear rather than discuss realities frankly; 

• Does not commit to a course of action unless it has been planned and researched near 
perfection; 

• Plays it safe rather than risk making mistakes; and 

• Waits for others to take the lead or agree in challenging situations. 

Effective Behaviors 

• Addresses difficult issues, puts self on the line to deal with problems; 

• Commits resources and/or time in the face of uncertainty to reach a challenging goal; 

• Displays strong but realistic beliefs in own capabilities and ideas; and 

• Acts decisively even with incomplete information and assumes responsibility for the 
outcome. 

Very Effective Behaviors 

• Questions organizational norms and accepted thinking and practices and champions better 
ideas for Ford Motor Company and the consumer; 

• Takes on the task of communicating unpleasant feedback/information to colleagues; 

• Supports and defends staff for taking calculated risks in order to transform and grow the 
business; and 

• Takes bold, decisive actions despite risks, conflict or uncertainty; also takes on calculated, 
entrepreneurial risks and assumes accountability for outcomes. 



Raters are also asked to consider actions not taken, as well as those taken. Raters are advised to 
avoid allowing initial impressions to bias their ratings, and they are not supposed to rely on their 
most recent encounter with someone when they are making their assessments. Raters are 
cautioned to be careful not to make generalizations from one aspect of a person’s behavior to all 
aspects; rather, they are to examine each behavior individually. The behavior examples provided 
for each particular level of performance for each competency can really help raters provide 
informed assessments. They are asked to study the examples given and to consult them as they 
make their own judgments. 

The employee and his or her manager closely review the final leadership assessment report. This 
information is used to help plan each individual’s professional development. Together they 
review the relative strengths and weaknesses of performance, the behaviors where raters indicate 



2 Material copied with permission of John M. Raushenbarger, Ford Motor Company. 



171 



181 



consensus, behaviors where raters’ opinions differ significantly, behaviors where self-assessment 
is substantially different that those of other raters, and leadership behaviors that the manager and 
employee identify as the most critical to the employee’s position. Ultimately, the manager and 
the employee will consider how the report can be used to plan the employee’s professional 
development. It will also suggest how to improve upon specific leadership behaviors. Finally, 
the manager must integrate the feedback into the overall evaluation of each employee. 

This information is encoded and sent to an outside vendor for consolidation and reporting. Then 
the report is sent on a confidential basis to the employee and his or her manager. Finally, the 
employee and his or her manager review copies of the Leadership Assessment Report and other 
relevant measures of performance. The manager uses this information to assess the employee’s 
overall performance. The employee and manager jointly identify appropriate actions for the 
future. 



Data Ramifications 

The screening assessments to examine the skills and knowledge of potential employees are 
strengthened since there is a rigorous training process for all raters. This process helps to 
increase the reliability of ratings so there is more consistency across raters. In addition, each 
level of performance is clearly anchored with an example that helps raters to better understand the 
dimensions of performance associated with different levels. The validity of the assessment 
results seem fairly strong, since the problems reflect open-ended issues that capture the realities 
of the actual job. A clear benefit for potential employees is that they get a better understanding 
about what is important at Ford. The staff at Ford get more objective results, since all candidates 
are taking the same assessments. The use of multiple raters for each group of individuals also 
makes the assessments more reliable. 

Ford staff hope in the future to establish predictive validity so that they can accurately predict 
who will perform most effectively in certain types of positions. They want to learn how to best 
match employees with certain jobs. As the assessments are used over time and data are 
evaluated, the staff want to be able to predict which employees would work best in certain 
positions. This is an ambitious goal that will take considerable time and expertise to achieve. 
However, clearly they have the expertise at Ford, and with more time to implement their new 
assessments, they will move in this direction. 



Advantages and Challenges Associated with a Competency-Based System 

One of the major challenges associated with this particular system is the enormous amount of 
time that it takes to conduct the performance assessments. It is a very hard process to manage, 
given the high numbers of individuals who are assessed by at least two raters. In addition, it is 
very expensive to implement. However, the leadership at Ford believes that the expenses are well 
worthwhile because they enable the company to hire the very strongest candidates possible. 
While campus interviews are the easiest to conduct, most individuals at Ford strongly believe the 
second layer of assessments conducted by the assessment centers are important to maintain as 
requirements for all prospective candidates. However, Ford is beginning to explore the 
possibility of offering some of the performance assessments as Web-based tools. These are early 
thoughts, but there is some serious concern with the amount of time (3 to 4 hours) that candidates 
spend completing their assessments. Each assessor spends a full day doing the evaluations and 
writing up his/her results for each small group of candidates. 
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The advantages associated with competencies include the belief that all units at Ford have a 
common understanding about them since they all use a common language across functions and 
units. Then assessments can be linked to these competencies to determine the levels of 
performance by employees. 



Recommendations for Other Colleges and Universities 

Individuals at Ford strongly encourage colleges to build a strong competency-based education 
model that specifically articulates the important skills and knowledge that students need to be 
effective. A strong model will be complex, but it is essential to embed these competencies across 
layers of the organization. When the competencies are embedded, then there are strong 
reinforcements of particular skills and knowledge across layers of the organization that help 
employees build stronger skills. Ideally, corporations should work more closely with colleges 
and universities to identify these important competencies. 



Conclusion 

The Ford Motor Company has established specific competencies that cut across all layers of the 
organization. Individuals seem to have a clear understanding about these competencies and they 
are directly aligned with formal assessments that are conducted at multiple points in time. Ford 
Motor Company has the advantage of being able to work with researchers and industrial 
organizational psychologists who can provide the necessary expertise to design assessments and 
then create feedback reports. The assessments have strong reliability, and validity should become 
more apparent as they are used over time. 
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APPENDIX I 



CASE STUDY INTERVIEW PROTOCOL 



Thank you for meeting with me today. As you know, I am interviewing you as part of a NPEC 
[explain NPEC if needed] project studying the data ramifications of competency-based initiatives. 

1 . What is your name and position? How have you been involved in [NAME OF 
COMPETENCY-BASED INITIATIVE]? 

2. Describe how competencies are used in the [NAME OF COMPETENCY-BASED 
INITIATIVE] process. 

Probe: Were competencies particularly useful? If yes, how? If no, why not? 

3. How were the competencies developed? 

Probe: Were competencies created from scratch or were they based on work done 
elsewhere? 

Probe: Are there any particular problems with “home-grown” competencies? 

Probe: At what level of specificity are the competencies expressed? Are they 
expressed numerically or qualitatively? Do they include cognitive and 
affective competencies? Attitudes or dispositions? 

4. Who provided leadership in the development of the competencies? What groups 
were involved? Do these groups remain responsible for the competencies? 

5. How are the reliability and validity of these competencies ensured? 

6. How are you measuring the identified competencies? 

Probe: What types of assessments are used? 

7. Are competencies linked structurally throughout the institution or organization? 

How? (For example, do program competencies guide course competencies?) 

8. Are these competencies transportable? How do you ensure transportability? 

9. Contrast the use of competencies with how the process was traditionally done. 

Probe: Is the competency-based process optional or required? 

10. What policies needed to be altered in order to accommodate using competencies? 

1 1 . What would you do differently if you were to start over? 

12. What recommendations would you offer to others who want to create competency- 
based learning experiences? 

13. Is there anything else you would like to share about this competency-based initiative? 
Thank you for your time. We appreciate your willingness to be interviewed. 
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